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PREFACE 


National  Transportation  Safety  Board  (NTSB)  accident  statistics 
reveal  that  a  large  portion  of  civil  aviation  accidents  are  in 
part  related  to  poor  flying  judgment.  In  recognition  of  the 
seriousness  of  this  problem,  the  Federal  Aviation  Administration 
(FAA)  in  1976  initiated  a  multi-phase  program  to  improve  the 
judgment-making  abilities  of  civil  aviation  pilots.  The  first 
phase  of  the  program  resulted  in  a  report  by  researchers  at  the 
University  of  Illinois  which  contained  a  review  of  judgment- 
related  literature,  a  definition  of  pilot  judgment  and  the 
conclusion  that  the  training  and  evaluation  of  pilot  judgment  was 
feasible . 


The  Aviation  Research  Center  of  Embry-Riddle  Aeronautical 
University  (E-RAU)  in  October  of  1980  completed  a  second  phase  of 
the  pilot  judgment  study  for  the  FAA.  Course  materials  were 
developed  to  deliver  a  judgment  training  program  to  civil  aviation 
pilots.  The  program  was  designed  to  be  used  by  flight  instructors 
in  the  field  without  additional  support  materials. 


E-RAU  was  later  assigned  the  additional  tasks  of  optimizing  the 
judgment  training  materials  and  of  conducting  a  small-scale 
validation  experiment.  The  effort  was  conducted  under  the  Airmen 
Research  Program  in  general  aviation,  initiated  by  the  FAA 
Technical  center. 


This  volume  summarizes  the  implementation  and  the  results  of  the 
optimization  process  and  the  small-scale  evaluation  experiment 
performed  upon  optimir.ed  training  program  materials.  This  effort 
was  carried  out  by  the  Aviation  Research  Center  of  Embry-Riddle 
Aeronautical  University  under  contract  number  DOT-FA-79NA-6040 . 
Dr.  Alan  Diehl  was  the  Contracting  Officer's  Technical 
Representative  for  the  FAA's  Office  of  Aviation  Medicine.  The 
project  was  also  monitored  by  the  FAA's  Office  of  Flight 
Operations . 


The  contents  of  this  three  volume  document  are  based  on  work  which 
was  accomplished  during  E-RAU' s  initial  effort  and  which  was 
conceived  and  conducted  under  the  guidance  of  Dr.  Jerome  I. 
Berlin,  then  Director  of  Aviation  Research  Center  at  E-RAU. 
Original  versions  of  the  Student  Manual  and  the  Instructor  Manual 
were  conceptualized  and  written  during  this  effort  by  a  team  of 
scientists  at  the  E-RAU  Aviation  Research  Center  with  consulting 
assistance  from  Social  Science  Research  Associates  (SSRA) . 
Authors  included  Dr.  Berlin  ol  E-RAU,  Mr.  Eric  V.  Gruber  (E-RAU), 
Dr.  James  W.  Mills  (SSRA),  Dr.  Phillip  K.  Jensen  (SSRA),  Dr.  James 
M.  O'Kane  (SSRA)  and  Dr.  Charles  W.  Holmes  (E-RAU). 


The  optimization  of  the  two  manuals  and  the  evaluation  experiment 
reported  here  were  conducted  by  a  team  of  Aviation  Research  Center 
personnel  at  E-RAU  under  the  direction  of  Dr.  Charles  w.  Holmes. 
Data  collection  was  conducted  principally  by  Mr.  James  R.  Lau. 
Principle  editor  of  the  optimized  student  and  Instructor  Manuals 
was  Mr.  Eric  V.  Gruber,  with  assistance  from  Dr.  Holmes,  Mr.  Lau 
and  Mr.  Fred  Schwieg.  Assistance  with  training  portions  of  the 
evaluation  experiment  was  provided  by  E-RAU  flight  instructors 
Mr.  Niels  Christensen,  Mr.  Jim  Zurales  and  Mr.  Mike  Shephard. 

This  Technical  Summary  was  prepared  by  Dr.  Holmes,  Mr.  Lau,  and 
Mr.  Gruber.  Drs.  James  W.  Mills,  Phillip  K.  Jensen  and  James  M. 
O'Kane  of  SSRA  provided  consultation  regarding  the  statistical 
analyses . 

USER  COMMENTARY  . 


These  pilot  judgment  training  program  materials  are  both  new  and 
innovative.  It  is  hoped  that  user  experiences  and  commentary  will 
be  reported  to  the  FAA  offices  involved  in  this  research.  Please 
address  your  written  commentary  to  one  or  both  of  the  FAA  offices 
listed  below. 


Dr.  Alan  Diehl 

Program  Scientist  -  Human  Performance 
Federal  Aviation  Administration/AAM  540 
Washington,  D.C.  20591 


Mr.  Douglas  P.  Harvey 

Manager  -  Airmen  Research  Program 

Federal  Aviation  Administration  Technical  Center/ATC-340D 
Atlantic  City  Airport,  New  Jersey  08405 
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EXECUTIVE  SUMMARY 


National  Transportation  Safety  Board  accident  statistics 
(reference  1)  reveal  that  a  large  portion  cf  civil  aviation 
accidents  are  related  to  poor  flying  judgment  as  a  primary  or 
contributing  cause.  Therefore,  a  multi-phase  research  program  was 
initiated  by  the  Federal  Aviation  Administration  (FAA)  in  1976  to 
answer  questions  concerning  the  definition,  nature  and  possible 
means  for  improvement  of  pilot  judgment  in  general  aviation. 
Researchers  at  the  Aviation  Research  Laboratory  of  the  University 
of  Illinois  completed  an  initial  study  in  1977.  This  study 
provided  a  comprehensive  review  of  literature  and  produced  a 
definition  of  pilot  judgment.  The  study  concluded  that  the 
improvement  and  the  evaluation  of  pilot  judgment  was  feasible,  and 
it  described  a  broad  approach  for  teaching  and  evaluating  pilot 
judgment. 

Embry-Riddle  Aeronautical  University  (E-RAU)  was  assigned  the  task 
of  developing,  optimizing  and  evaluating  judgment  training 
materials  for  a  small-scale  validation  experiment,  using  student 
pilots  and  flight  instructors  affiliated  with  the  University. 
This  document  outlines  the  concepts  upon  which  the  training 
materials  are  based,  it  describes  the  optimization  process  which 
yielded  the  present  Student  and  Instructor  Manuals  (volumes  II  and 
III  of  this  document) ;  and  it  summarizes  the  method  and  results  of 
the  experiment  performed  to  evaluate  their  effectiveness. 

PURPOSE. 

The  purposes  of  the  study  were: 

1.  to  optimize  the  training  materials  by  improving  the 
presentation  of  the  information  while  maintaining  or  improving  the 
learning  results; 

2.  to  demonstrate  the  effectiveness  of  the  program  in  a 
small-scale  experiment;  and 

3.  to  develop,  and  to  test  on  a  small  scale,  methods  and 
procedures  for  the  objective  measurement  of  pilot  judgment. 


CONCEPTS  AND  IMPLEMENTATION. 

The  course  materials  are  contained  in  two  volumes.  The  Student 
Manual  (volume  II)  contains  18  lessons  of  instructional  material 
and  companion  worksheets.  The  Student  Manual  presents  new  terms 
and  concepts  which  have  been  especially  designed  to  lead  the 


ix 


student  into  modified  patterns  of  thinking.  These  include  the 
definition  of  pilot  judgment,  "the  three  subject  areas,"  "the  six 
action  ways,"  "the  poor  judgment  sequence  chain,"  and  "the  three 
mental  processes  of  safe  flight."  In  addition,  newly  derived 
behavioral  aspects  of  pilot  judgment  are  introduced.  These 
include  "the  five  hazardous  thought  patterns,"  "antidotes"  for  the 
hazardous  thoughts,  and  techniques  for  identifying  and  reducing 
stress . 

The  Instructor  Manual  (volume  III)  reviews  the  contents  of  the 
Student  Manual,  describes  the  instructional  delivery  of  the 
program's  conceptual  and  behavioral  aspects,  and  presents  two  sets 
of  lessons  for  the  instructor  to  conduct  during  flight  training. 
It  also  contains  instructional  management  materials  for  the 
individual  flight  instructor  as  well  as  the  training  supervisor. 

The  "Concepts  and  Implementation"  section  of  this  Technical 
Summary  specifies  each  new  concept  and  behavioral  aspect  that  the 
training  materials  contain.  Two  learning  theories,  behavior 
modification  and  facilitation,  were  used  in  the  program's  design. 
An  outline  of  these  two  theories  and  a  discussion  of  their  role  in 
the  implementation  of  the  instruction  are  also  presented  in  this 
sect  ion . 

OPTIMIZATION. 

APPROACH .  Optimization  of  the  Student  Manual  was  conducted  in  two 
stages  using  small  groups  of  E-RAU  flight  students.  During  the 
first  stage,  three  students  read  ;  nd  reviewed  the  manual.  In  the 
second  stage,  six  students  performed  a  similar  review.  Modifica¬ 
tions  and  revisions  were  made  following  each  iteration  in  response 
to  the  deficiencies  uncovered.  A  third  iteration  was  conducted 
concurrently  with  the  evaluation  experiment  and  was  performed  by 
the  27  subjects  in  the  experimental  group.  However,  only  minor 
changes  were  made  following  the  exercise. 

Optimization  of  the  Instructor  Manual  was  accomplished  in  two 
stages.  First,  two  E-RAU  senior  flight  instructors  read  and 
critically  reviewed  the  manual.  This  information,  plus  changes 
made  to  the  Student  Manual  during  its  second  optimization, 
provided  the  basis  for  a  first  revision.  The  second  stage  was 
conducted  concurrently  with  the  evaluation  experiment,  and  minor 
revisions  were  made  based  on  information  gathered  from  the  three 
flight  instructors  involved  in  that  effort. 
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RESULTS .  The  Student  Manual  underwent  alterations  throughout  its 
content.  Eleven  major  changes,  described  in  the  body  of  this 
report,  were  incorporated  and  the  time  required  for  the  average 
student  to  complete  the  manual  was  verified  to  be  just  over  seven 
hours.  The  Instructor  Manual  also  underwent  major  revisions,  and 
seven  major  changes  were  incorporated.  An  additional  unit 
containing  necessary  items  to  support  the  instructor's  management 
of  the  instruction  was  added,  and  conformity  was  maintained  to  the 
final  optimized  Student  Manual. 


EVALUATION. 


METHOD .  The  experimental  approach  consisted  of  three  groups  of  27 
subjects  each.  The  expor imenta 1  group  received  an  academic 
pretest,  judgment  ground  and  flight  training,  an  academic 

posttest,  and  an  observation  flight  at  the  completion  of  the 
training  program.  The  fliqht  control  group  received  only  the 
observation  flight.  The  academic  control  group  received  the  same 
written  pretest  and  posttest  as  the  experimental  group. 

The  subjects  were  all  students  at  Embry-Riddle  Aeronautical 
University  (E-RAU)  Daytona  Beach,  Florida.  They  were  randomly 

selected  from  volunteers  enrolled  in  the  University's  FA  103— 

Basic  Flight  course,  which  includes  students  who  have  met  the 
requirements  for  solo  flight  and  are  engaged  in  flight  training 
leading  to  the  attainment  of  the  private  pilot  certificate. 

Attrition  for  the  three  groups  was  minimal,  with  26  subjects 
completing  th<*  activities  from  the  experimental  group,  24  from  the 
flight  control  group  and  25  from  the  academic  control  group. 

Two  instructors  were  employed  and  trained  by  the  E-RAU  Aviation 
Research  Center  project  team  to  celiver  the  training  program  to 
the  experimental  group  and  to  conduct  the  observation  flights 
which  followed  it. 

Judgment  training  for  the  experimental  group  was  completed  ii.  x 

approximately  three  weeks.  Ground  training  consisted  of  study  of  \ 

the  Student  Manual  with  guidance  from  the  students'  judgment 
program  flight  instructor.  Flight  training  consisted  of  three 
2-hour  flights  during  which  the  judgment  program  flight  instructor 
administered  in-flight  exercises  from  the  Instructor  Manual  units 
IV  and  V. 

DATA  COLLECTION.  Data  were  collected  from  the  two  types  of 
activities ,  a  written  pretest  and  posttest,  and  a  post-training 
observation  flight.  The  written  pretest  and  posttest  were 
identical  and  were  administered  to  the  experimental  group  and  the 
academic  control  group.  The  test  consisted  of  five  sections  which 
included  knowledge  of  general  pilot  judgment  concepts,  three  types 
of  pilot  activity  scenario  analyses,  and  recall  of  judgment 
terminology  presented  by  the  Student  Manual.  The  observation 
flight  was  of  approximately  one  and  three-quarters  hours  duration, 
and  it  was  administered  to  both  the  experimental  group  and  the 
flight  control  group.  All  subjects  were  directed  to  fly  a 
standardized  route  and  activity  scenario  by  the  flight  instructor 
administering  the  flight.  A  trained  observer  in  the  rear  seat  of 
the  aircraft  observed  the  subjects  during  20  specific  judgment 
situations  for  good  judgment  or  poor  judgment  responses.  At  the 
conclusion  of  the  observation  flight,  subjects  in  both  groups 
completed  a  short  interview  which  provided  subjective  data 
regarding  experiment  participation.  Subjects  in  the  experimental 
group  also  answered  questions  regarding  the  effectiveness  of  the 
Pilot  Judgment  Training  Program.  i 
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RESULTS .  Data  from  written  testing  and  the  observation  flights 
indicated  statistically  significant  differences  between  the 
performances  of  the  experiments]  group  and  control  group  subjects. 

For  the  written  tests,  results  from  each  of  the  five  sections  were 
analyzed  using  analysis  of  variance  (ANOVA) .  Results  of  the 
analysis  indicated  the  scores  of  the  experimental  croup  were 
significantly  higher  than  those  of  the  control  group  on  all 
sections  of  the  posttest  found  to  be  valid  and  relevant.  Even  the 
portion  of  the  test  with  general  pilot  judgment  concepts  that 
avoided  the  specific  terminology  of  the  program  indicated  higher 
scores  for  the  experimental  group.  The  results  for  the  section 
dealing  with  factor  importance  ratings  were  not  analyzed  since 
this  section  was  found  to  have  questionable  validity  as  a 
measurement  instrument. 

Analysis  of  the  observation  flight  was  conducted  using  a  one-way 
analysis  of  variance.  The  results  of  this  test  were  again 
statistically  significant  (p  <0.001)  meaning  that  the  probability 
is  less  than  one  in  one  thousand  that  the  difference  is  due  to 
chance  alone.  The  experimental  group  demonstrated  a  markedly 
better  performance  on  the  post- training  observation  flight  than 
did  the  control  group. 

Subjective  data  collected  in  the  interviews  following  the 
observation  flights  indicated  generally  favorable  reactions  from 
subjects.  All  subjects  from  the  experimental  group  indicated  a 
belief  that  the  training  would  be  of  value  to  them  in  their  future 
flying,  and  many  mentioned  that  they  had  recognized  poor  judgment 
situations  during  flying  done  outside  the  judgment  program. 

DISCUSSION.  The  data  indicated  statistically  significant 
differences  between  the  performance  cf  experimental  and  control 
group  subjects.  This  was  true  both  in  the  acquisition  of  judgment 
concepts  and  in  the  skills  performance  as  measured  during  judgment 
flight  situations.  Tne  experimental  group  evidenced  an  increase 
in  knowledge  of  the  judgment  concepts  while  the  control  group 
showed  no  significant  change.  Also,  the  experimental  group 
evidenced  a  markedly  better  performance  on  the  post-treatment 
judgment  skills  observation  flight  than  did  the  control  group. 

One  of  the  major  tenets  of  the  program  is  that  students  must 
master  judgment  concepts  before  those  concepts  can  be  applied  in 
judgment  flight  situations.  Concepts  of  the  judgment  training 
were  not  only  taught  during  the  ground  school,  but  were  constantly 
reinforced  during  flight  training  and  during  preflight  and 
postflight  briefings.  Although  it  cannot  be  said  that  a  direct 
causal  relationship  was  proven  to  exist  between  the  judgment 
ground  training  and  the  improvement  of  judgment  in  flight  as 
measured  by  the  observation  flight,  it  appears  evident  that  the 
integrated  ground  training  and  flight  training  did  have  a  positive 
effect  on  subject  performance  in  judgment  flight  situations. 
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The  subjects  participating  in  the  experimental  group  held  strongly 
positive  attitudes  about  the  benefits  of  the  judgment  training 
program  at  the  experiment's  conclusion.  A  substantial  majority  of 
the  group  responded  favorably  to  all  the  questions  posed  by  the 
attitude  questionnaire.  They  reported  that:  1)  they  would  pay 
greater  attention  to  judgment  concepts  in  their  own  future  flying, 
2)  they  believed  the  program  to  be  "a  good  idea,"  and  3)  they 
would  like  to  see  judgment  training  material  be  part  of  the 
required  studies  for  a  private  pilot  license. 

Two  potential  limitations  to  the  general izability  of  the  program's 
potential  effects  as  indicated  by  these  results  are  noted.  First, 
the  subjects  involved  were  a  relatively  homogeneous  group.  This 
gives  rise  to  the  question  as  to  how  the  program  might  transfer  to 
a  more  heterogeneous  group  of  subjects.  Second,  the  training 
period  was  compressed,  covering  only  one  month.  A  conclusion 
cannot  be  drawn  as  to  how  the  program  might  function  when 
integrated  into  the  training  schedule  of  an  airport  based  flight 
school  where  the  training  period  for  student  pilots  usually  ranges 
from  six  to  twelve  months. 


CONCLUSIONS. 

Results  of  the  experiment  indicate  that  the  implementation  of  the 
optimized  manuals  had  a  positive  effect  on  the  judgment-making 
abilities  of  subjects  in  the  experimental  group,  as  measured  by  an 
observation  flight  and  a  written  test.  It  is  hypothesized  that 
similar  results  would  be  obtained  in  the  overall  general  aviation 
population,  resulting  in  improved  pilot  judgment-making  abilities. 


INTRODUCTION 


BACKGROUND. 


National  Transportation  Safety  Board  accident  statistics 
reveal  that  a  large  portion  of  civil  aviation  accidents  are  in 
part  related  to  poor  flying  judgment.  During  the  period  from  1973 
to  1977  the  pilot-in-command  was  listed  as  the  cause  or  a  related 
factor  in  S3  percent  of  all  general  aviation  accidents  and  as  the 
cause  or  a  related  factor  in  88  percent  of  the  fatal  accidents 
that  occurred  during  the  same  period  (reference  1).  Since  these 
causal  statistics  may  include  the  influence  of  more  than  one 
factor,  it  is  at  best  difficult  for  the  researcher  to  identify 
accidents  which  were  caused  solely  by  pilot  "judgment." 

Another  difficulty  confronting  serious  attempts  to  investigate  the 
judgment  aspects  of  pilot  behavior  has  been  the  lack  of  a  precise, 
functional  definition  of  "pilot  judgment."  Further,  there  has 
been  uncertainty  about  whether  or  not  a  pilot's  judgment  abilities 
can  be  improved  through  training. 

Recognizing  the  continuing  safety  issue  of  poor  pilot  judgment,  as 
well  as  its  associated  difficulties  and  uncertainties,  the  Federal 
Aviation  Administration  ( FAA)  initiated  a  multi-phase  research 
program  in  1976  to  seek  answers  to  the  problem.  Undertaking  the 
first  phase  of  research,  the  Aviation  Research  Laboratory  of  the 
University  of  Illinois  conducted  a  study  entitled.  Judgment 
Evaluation  and  Instruction  in  Civil  Pilot  Training  (reference  2). 
That  study  contained  a  comprehensive  review  of  literature  relating 
to  judgment  training  •  a  aviation  and  other  related  fields.  In 
addition,  the  study  produced  the  following  results: 

1.  a  definition  of  pilot  judgment; 

2.  a  conclusion  that  the  improvement  and  the  evaluation  of  a 
pilot's  judgment  is  feasible;  and 

3.  the  development  of  a  broad  approach  for  teaching  and 
evaluating  judgment. 

With  th<-  work  of  the  University  of  Illinois  providing  background 
information  and  guidance,  the  Aviation  Research  Center  of 
Embry-Riddle  Aeronautical  University  (E-RAU)  completed  a 
second-phase  study.  This  effort  resulted  in  innovative  training 
materials  for  improving  the  judgment-making  abilities  of  primary 
flight  student  pilots,  as  well  as  a  systematic  approach  for  use  by 
flight  instructors  to  administer  the  judgment  training. 
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PURPOSE. 


The  current  study  had  three  major  purposes.  The  first  was  to 
develop  optimized  pilot  judgment  training  and  evaluation  course 
materials.  The  optimization  procedure  was  oriented  toward 
improving  the  presentation  of  the  information,  while  maintaining 
or  improving  the  learning  results.  A  second  purpose  was  to 
demonstrate  the  effectiveness  of  the  program  in  a  small-scale 
experiment.  It  was  not  the  intent  of  this  research  to  conduct  a 
full-scale  validation  of  the  Pilot  Judgment  Training  Program,  but 
to  provide  information  upon  which  a  decision  to  initiate  another 
phase  of  the  study  may  be  based.  The  third  purpose  was  to  develop 
and  to  test,  on  a  small  scale,  guidelines  and  procedures  for  the 
objective  measurement  of  pilot  judgment  in  both  ground  and  flight 
training  environments. 


CONCEPTS  AND  IMPLEMENTATION 


Using  new  concepts  and  constructs,  a  training  program  was  created 
which  endeavored  to  meet  the  following  terminal  objectives: 

1.  To  produce  a  program  that  was  practical,  easily  applied  and 
effective  in  producing  the  desired  response  from  the  students. 
The  contracting  office  of  the  FAA  specifically  discouraged  the  use 
of  elaborate  training  aids  such  as  audio-visual  materials  and 
simulators  during  this  early  phase  of  the  program. 

2.  To  design  the  program  so  that  it  would  achieve  meaningful 
integration  into  the  present  FAA  approved  ground  and  flight  school 
curriculum  and  so  that  it  would  not  significantly  impact  the 
amount  of  training  time  required  to  obtain  a  private  pilot 
license . 

3.  To  increase  the  student  pilot's  knowledge  and  understanding  of 
himself  by  teaching  him  to  better  recognize  and  identify  hazardous 
human  behavior  patterns. 

4.  To  teach  the  student  pilot  to  recognize,  analyze  and  evaluate 
factors  which  influence  judgment  in  regard  to  safe  flight. 

5.  To  train  the  student  pilot  to  recognize  and  to  cope  with 
potentially  hazardous  conditions  such  as  the  presence  of  hazardous 
thinking,  poor  judgment  chains  and  high  stress. 

6.  To  introduce  positive  changes  into  the  flight  instructor's  own 
teaching  behavior  and  attitudes. 

The  course  materials  are  contained  in  two  volumes.  The  Student 
Manual  contains  the  instructional  material  and  companion 
worksheets  which  provide  the  students  with  practice  activities  and 
progress  indicators.  The  Student  Manual  contains  18  lessons  which 
are  divided  into  three  units.  Unit  I  presents  terms  and  concepts 
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designed  to  both  develop  modified  patterns  of  thinking,  and  to 
give  students  and  instructors  the  most  objective  means  possible 
for  discussing  pilot  behavior.  Unit  II  addresses  behavioral 
aspects  of  judgment,  including  hazardous  thoughts  and  stress. 
Unit  III  relates  the  learning  of  units  I  and  II  to  actual  flight 
situations  using  scenarios  based  on  actual  pilot  accident  reports. 

The  Instructor  Manual  explains  how  the  flight  instructor  is  to 
present  judgment  training  to  the  students,  outlines  the  content  of 
the  Student  Manual,  presents  two  sets  of  lessons  for  the 
instructor  to  conduct  during  flight  training,  and  contains  support 
materials  for  the  instructor. 

PILOT  JUDGMENT  DEFINED . 

In  the  effort  to  capture  the  salient  aspects  of  pilot  judgment  in 
a  precise  definition,  judgment  in  and  of  itself  was  not  seen  as  a 
problem.  Rather,  it  was  the  ac tion s  that  pilots  took  based  upon 
their  poor  judgments  that  created  problems.  Since  the  training 
program  was  therefore  to  address  pilot  actions  (i.e.,  pilot 
judgment  behaviors) ,  it  was  first  necessary  to  establish  a 
specific,  functional  definition  of  pilot  judgment  in  this  context. 

The  National  Transportation  Safety  Board  (NTSBl  classifies 
aircraft  accidents  by  detailed  cause  factors.  Among  these  are 
factors  pertaining  to  judgment,  all  of  which  are  undfu  the  broad 
cause  of  "pilot"  and  include  such  quantifiable  factors  as  speed, 
distance,  altitude,  and  clearance.  In  the  context  used  in  the 
NTSB  accident  causes,  the  pilot  failed  to  estimate  one  or  more  of 
these  factors  correctly.  While  this  ability  to  estimate  the  value 
of  certain  factors  can  be  measured,  a  definition  of  "judgment” 
within  ttiis  context  is  too  limited  for  the  purpose  of  this  study. 

Jensen  and  Benel  (reference  2)  analyzed  all  accidents  occurring 
during  the  1970  -  1974  time  period  in  which  the  pilot  in  command 
was  listed  as  the  cause  or  a  related  factor.  They  classified 
these  pilot-caused  accidents  into  one  of  three  behavioral 
categories  --  procedural,  perceptual-motor ,  or  decisional 
activities.  They  found  that  a  majority  of  the  nonfatal 
pilot-caused  accidents  (56  percent)  were  the  result  of  faulty 
perceptual-motor  behavior.  This  category  included  such  factors  as 
failure  to  maintain  flying  speed  and  misjudgment  of  distance, 
speed,  altitude,  or  clearance.  A  majority  of  the  fatal 
pilot-caused  accidents  (51  percent)  resulted  from  faulty 
decisicnal  behavior.  They  found  the  most  significant  factors  in 
these  cases  to  be  the  familiar  "continued  VFP.  (visual  flight 
rules)  into  known  adverse  weather"  and  "inadequate  preflight 
planning  or  preparation."  It  became  clear  that  this  is  the  sort 
of  faulty  decisional  behavior  that  the  pilot  judgment  training 
program,  and  hence  the  operational  definition  of  judgment,  must 
address . 
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Webster's  New  Collegiate  Dictionary  (reference  3)  ,  defines 
judgment  as  " the  process  of  forming  an  opinion  or  evaluation  by 
discerning  and  comparing".  In  Dictionary  of  Behavioral  Science 
(reference  4),  judgment  is  defined  as  "a  critical  evaluation  of  a 
person,  object,  or  situation"  (p.  208).  English  and  English 
(reference  5)  define  judgment  as  "a  critical  evaluation  of  a 
person  or  a  situation.  It  includes  the  processes  of  appreciation, 
comparison,  and  appraisal  of  values"  (p.  282).  However,  S.I. 
Hayakawa  (reference  6)  describes  judgment  in  the  context  which  is 
best  suited  for  this  study.  Judgment,  according  to  Hayakawa,  is 
"sense  (his  emphasis)  applied  to  the  making  of  decisions, 
especially  correct  decisions"  (p.  69)  .  Hayakawa  describes  sense 
as  referring  to  "...rational  perception  accompanied  by  feeling. 
Used  this  way  it  suggests  an  intense  awareness  and  realization  of 
the  stimuli  to  which  it  is  responding  ...  The  word  is  commonly 
applied  to  the  ability  to  act  effectively  in  any  given  situation" 
(p.  690)  .  For  the  first,  time,  an  outcome  of  judgment  is  specified 
—  a  decision  to  act.  The  decision  is  made  based  upon  the 
awareness  and  realization  of  the  stimuli,  or  factors,  which  are 
forcing  the  decision.  Judgment,  therefore,  is  not  an  end;  it  is 
the  process  through  which  a  decision  is  made  to  take  some  sort  of 
action . 

In  making  the  decision,  the  pilot  must  consider  all  of  the  factors 
which  have,  or  should  have,  influence  upon  his  or  her  decision¬ 
making  process.  These  factors  shall  include  all  pilot,  aircraft, 
and  environmental  considerations.  Based  upon  these  considerations 
and  the  earlier  findings  of  Jensen  and  Benel’s  first-phase  study, 
pilot  judgment  is  defined  as  follows: 

Pilot  judgment  is  the  mental  process  by  which  the 
pilot  recognizes,  analyzes,  and  evaluates  informa¬ 
tion  regarding  himself,  the  aircraft,  and  the 
outside  environment.  The  final  step  in  the  process 
is  to  make  a  decision  pertaining  to  the  safe  opera¬ 
tion  of  the  aircraft  and  to  implement  the  decision 
in  a  timely  manner. 

Thus  it  can  be  said  that  if  a  pilot  properly  recognizes,  analyzes, 
and  evaluates  the  factors,  and  subsequently  makes  the  "proper" 
decision  and  implements  it  in  a  timely  manner,  then  he  or  she 
exercises  good  judgment.  If  he  or  she  does  not  properly 
recognize,  analyze,  or  evaluate  the  factors,  and  subsequently 
makes  a  poor  decision  which  leads  to  "improper"  or  untimely 
action,  then  he  or  she  has  exercised  poor  judgment. 

CONCEPTUAL  AND  BEHAVIORAL  ASPECTS  OF  TRAINING . 

The  Student  Manual  presents  new  terms  and  concepts  which  have  been 
especially  designed  to  lead  the  student  into  modified  patterns  of 
thinking.  In  addition,  newly  derived  behavioral  aspects  of  pilot 
judgment  are  introduced  to  the  student.  The  Instructor  Manual 
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contains  a  complete  presentation  of  the  intent  and  instructional 
delivery  of  these  conceptual  and  behavioral  aspects.  The 
following  section  specifies  the  program's  unique  instructional 
elements  to  give  the  reader  a  basic  understanding  of  the  training 
materials  which  are  discussed  within  this  volume. 

THE  THREE  SUBJECT  AREAS.  Each  poor  judgment  made  by  the  pilot  can 
be  categorized  to  involve  one  or  a  combination  of  three  specific 
subject  areas:  the  pilot  himself  (Pilot),  the  aircraft  and  its 
subsystems  (Aircraft) ,  and  the  outside  environment  (Environment) . 

The  Pilot  subject  area  concerns  the  pilot's  state  of  health, 
competency  in  a  given  situation,  level  of  fatigue  and  any  other 
factors  that  may  effect  his  performance. 

The  Aircraft  subject  area  includes  consideration  with  respect  to 
the  aircraft's  airworthiness,  its  powerplant,  and  its  equipment  as 
well  as  performance  criteria  such  as  weight  and  balance  and  runway 
requirements . 

The  Environment  subject  area  concerns  not  only  items  such  as  the 
weather,  take-off  conditions,  airfield  altitude  and  temperature, 
but  also  outside  information  inputs  such  as  weather  briefings  (or 
lack  thereof),  ATC  instructions,  and  other  considerations. 

THE  SIX  ACTION  WAYS.  Nearly  fcUO  NTSB  accident  briefs  were 
examined  to  determine  how  pilots  carry  out  the  actions  resulting 
from  their  decisions.  It  became  obvious  that  pilots  implemented 
their  decisions  in  six  ways,  and  that  these  "action  ways"  could  be 
grouped  in  three  pairs: 

1.  DO  -  NO  DO 

2.  UNDER  DO  -  OVER  DO 

3.  EARLY  DO  -  LATE  DO 

The  actions  ways  are  defined  as  follows; 

1.  DO:  The  pilot  did  something  he  or  she  should  not  have  done. 

2.  NO  DO:  The  pilot  did  not  do  something  he  or  she  should  have 
done. 

3.  UNDER  DO:  The  pilot  did  not  do  enough  when  he  or  she  should 
have  done  more. 

4.  OVER  DO:  The  pilot  did  too  much  when  he  or  she  should  have 
done  less. 
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5.  EARLY  DO:  The  pilot  acted  too  early  when  he  or  she  should 
have  delayed  acting. 

6.  LATE  DO:  The  pilot  acted  too  late  when  he  or  she  should  have 
acted  earlier. 

These  action  ways  are  associated  only  with  poor  judgment.  In 
order  to  exercise  good  judgment  it  would  be  necessary  for  the 
pilot  to  perform  the  action  required  and  only  that  action,  perform 
the  action  only  to  the  degree  necessary,  and  to  perform  the  action 
on  time. 

THE  POOR  JUDGMENT  BEHAVIOR  CHAIN.  Ontiveros,  Spangler,  and  Sulzer 
(reference  7)  stated  that  most  accidents  result  from  a  combination 
of  circumstances,  rather  than  a  single  cause  such  as  pilot  error, 
aircraft  defect,  or  environmental  stress.  They  also  emphasized 
that  an  aircraft  accident  is  the  end  result  of  this  causal  chain. 
Similarly,  it  can  be  said  that  most  accidents  do  not  result  from 
one  error  in  judgment.  An  accident  would  be  more  likely  to  occur 
as  a  result  of  a  series  of  errors  in  judgment  pertaining  to  pilot 
proficiency  or  experience,  aircraft  condition,  or  stress  caused  by 
the  outside  environment.  This  series  of  errors  in  judgment  is 
called  the  poor  judgment  behavior  chain  (PJ  chain) . 

It  is  impossible  for  accident  investigations  by  the  NTSB  to 
precisely  establish  the  PJ  chain  of  behaviors  which  lead  to  a 
pilot  being  involved  in  an  accident.  This  is  especially  true  in 
fatal  accidents.  It  is  therefore  impossible  to  establish  a 
pattern  for  the  PJ  chain  on  an  empirical  basis.  The  following 
principles  of  the  PJ  chain  are  therefore  a  priori. 

1 .  One  poor  judgment  increases  the  probability  that  another  poor 
judgment  will  follow^  Since  judgments  are  made  on  information 
about  oneself,  the  aircraft,  or  the  environment,  the  pilot  is  more 
likely  to  make  a  poor  judgment  if  the  input  factors  are  not 
accurate.  One  poor  judgment  provides  an  erroneous  bit  of 
information  which  the  pilot  must  consider  when  making  subsequent 
judgments . 

2 .  The  more  poor  judgments  made  in  sequence,  the  more  probable 
that  others  will  continue  to  follow.  The  reasoning  for  this 
principle  Is  the  same  as  that  In  the  previous  principle,  except 
that  it  is  concerned  with  multiple  poor  judgments  in  sequence. 
The  more  erroneous  information  used  by  the  pilot  to  mak“ 
judgments,  the  more  likely  it  is  that  the  pilot  will  make 
subsequent  poor  judgments. 

3 .  As  the  PJ  chain  grows,  the  alternatives  for  safe  flight 
decrease .  ft  is a  priori  that  if  a  pilot  selects  one  alternative 
among  several,  the  option  to  select  the  remaining  alternatives  may 
be  lost.  Ecr  example,  if  a  pilot  makes  a  poor  judgment  to  fly 
through  a  hazardous  weather  area,  the  alternative  to 
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circumnavigate  the  weather  is  lost  once  severe  weather  is 
encountered. 

4 .  The  longer  the  PJ  chain  becomes,  the  more  probable  it  is  that 
disaster  will  occur"  As  the  PJ  chain  grows  longer,  fewer  and 
fewer  alternatives  For  safe  flight  are  available  to  the  pilot.  As 
the  alternatives  for  safe  flight  become  fewer,  the  greater  the 
chance  becomes  that  an  accident  will  occur. 

It  is  imperative  that  the  PJ  chain  be  broken  as  early  as  possible. 
Since  pilot  judgment  is  defined  as  a  mental  process  by  which 
pilots  recognize,  analyze,  and  evaluate  information,  it  is 
reasonable  to  assume  that  pilots  can  be  trained  in  the  process. 

BREAKING  THE  PJ  CHAIN.  The  single  most  important  first  step  in 
breaking  the  PJ  chain  is  recognition  by  the  pilot  that  he  or  she 
has  made  a  poor  judgment.  If  recognition  of  the  poor  judgment  is 
not  made,  there  is  little  chance  to  prevent  the  subsequent 
recurrence  of  poor  judgments.  In  order  to  recognize  that  a  poor 
judgment  has  been  made,  the  pilot  must  receive  evaluative  or 
corrective  information  about  his  or  her  judgment  process  He  or 
she  must  be  able  to  perceive  the  results  of  actions  )  ased  on 
previous  decisions,  and  use  this  information  as  input  factors  in 
subsequent  judgment  processes.  The  pilot  must,  in  other  words, 
recognize  and  use  feedback .  A  pilot  generally  receives  feedback 
from  two  sources: 

1.  The  pilot's  own  senses,  and 

2,  An  outside  observer. 

Since  good  judgment  is  a  learned  process,  it  is  generally 
necessary  that  feedback  initially  comes  from  an  outside  observer 
(i.e.,  the  instructor).  As  judgment  training  continues,  the  pilot 
will  learn  to  obtain  feedback  from  his  or  her  own  senses.  Prior 
interpretation  of  feedback  is  sometimes  difficult  in  that  the 
pilot  is  often  hesitant  to  admit  that  he  or  she  has  made  an  error 
in  judgment.  Yet,  recognition  of  poor  judgment  is  necessary  in 
order  to  break  the  PJ  chain  as  quickly  as  possible.  The  following 
steps  are  prescribed  in  order  to  help  pilots  break  the  PJ  chain: 

1.  Recognize  that  a  poor  judgment  has  been  made  (utilize 
feedback).  Admit  the  error  in  judgment. 

2.  Check  for  personal  stress  that  could  allow  the  PJ  chain  to 
continue . 

3.  Engage  in  problem  resolving  (PR)  to  correct  the  problem  that 
resulted  from  the  poor  judgment. 
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4.  Search  for  other  poor  judgments.  The  pilot  must  remember 
that  poor  judgments  tend  to  occur  in  chains,  and  must  be  sure  that 
he  or  she  has  broken  the  PJ  chain. 

5.  Diagnose  the  original  poor  judgment  to  provide  oneself  the 
feedback  needed  to  avoid  making  a  similar  poor  judgment  in  the 
future . 

THE  THREE  MENTAL  PROCESSES  OF  SAFE  FLIGHT.  The  mental  processes 
as  used  here  are  applied  to  safe  flight.  However,  they  may  in 
fact  be  applied  to  any  activity  or  process.  People  do  certain 
things  automatically,  they  solve  problems,  and  they  review 
on-going  processes  .in  order  to  assess  their  current  status.  The 
three  mental  processes  are  defined  as  follows: 

1.  Automatic  Reaction  (AR)  is  the  mode  of  thinking  that  allows 
the  pilot  to  maintain  control  of  the  aircraft  while  simultaneously 
engaging  in  other  activities.  The  pilot  maintains  heading  and 
altitude  by  making  small,  automatic  adjustments  in  power  and 
attitude.  The  pilot  can  simultaneously  perform  other  tasks  such 
as  talking  on  the  radio  and,  in  the  case  of  the  proficient  pilot, 
taking  care  of  in-flight  emergencies.  The  pilot  finds  that,  with 
practice,  the  need  to  "think  about"  what  to  do  will  decline  and 
eventually  be  eliminated  as  his  or  her  skills  become  Automatic 
Reactions . 

2.  Problem  Resolving  (PR)  is  the  mode  of  thinking  that  helps  a 
pilot  overcome  undesirable  situations  by  means  of  a  systematic 
process.  The  systematic  process  takes  place  in  three  steps.  In 
step  cne,  the  pilot  recognizes,  analyzes,  and  defines  a  problem. 
In  step  two,  the  pilot  considers  the  methods  to  solve  the  problem 
and  the  possible  outcomes  of  the  possible  solutions.  In  step 
three,  the  pilot  applies  the  selected  solution  to  the  best  of  his 
or  her  ability.  In  problem  resolving,  one  must  work  through  a 
process;  in  automatic  reaction,  one  just  does . 

3.  Repeated  Reviewing  (RR)  is  the  mode  of  thinking  that  allows 
the  pilot  to  be  continuously  aware  of  all  the  factors 
(pilot/aircraft/environment)  that  affect  safe  flight.  The  pilot 
is  continuously  trying  to  anticipate  which  factor  will  require 
that  his  or  her  mental  activities  be  engaged  in  PR  or  AR.  By 
repeated  reviewing,  the  pilot  can  keep  an  awareness  of  all  the 
conditions,  from  weather  conditions  to  aircraft  performance  to  the 
pilot's  own  state  of  health,  that  contribute  to  safe  flight  or 
have  tne  potential  of  leading  to  disaster. 

THE  FIVE  HAZARDOUS  THOUGHT  PATTERNS.  Jensen  and  Benel  (reference 
2)  stated  that  every  decision  that  a  pilot  makes  is  influenced  by 
physiological,  psychological  and  social  pressures  which  are 
virtually  impossible  to  measure  at  the  time  the  decision  is  made. 
In  addition,  self-image  and  the  need  to  maintain  that  image 
externally  can  effect  the  pilot's  judgment.  The  question 
remained  as  to  the  identity  of  thought  patterns  which  accompany 
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the  type  of  self-image  and  the  type  of  external  image  manifested 
by  the  pilot  displaying  poor  judgment.  It  was  necessary  to 
identify  specific  thought  patterns  which  would  make  a  pilot 
willing  to  violate  regulations,  extend  safety  margins,  exceed 
legal  limitations,  or  attempt  to  operate  an  aircraft  in  conditions 
beyond  his  or  her  capabilities.  In  other  words,  what  are  the 
thought  patterns  which  cause  pilots  to  exhibit  what  Jensen  and 
Benel  have  labeled  "irrational  pilot  judgment?" 

It  was  r^tulated  that  if  these  thought  patterns  could  be 
identified,  then  pilots  could  be  trained  to  recognize  them  in 
their  own  thinking  and  to  apply  corrective  actions.  Little  prior 
research  was  found  in  which  such  thought  patterns  were  described. 
Thus,  it  was  found  appropriate  to  consult  experts  in  the  psycho¬ 
logical  and  sociological  sciences  to  obtain  informed  opinions  on 
the  nature  of  such  hazardous  thought  patterns.  This  resulted  in 
the  identification  of  five  thought  patterns  and  the  assigning  of 
descriptive  names  for  those  thoughts.  Descriptions  of  the 
thoughts  which  were  identified  are  contained  in  the  following 
paragraphs . 

1.  Anti-authority .  This  is  the  thought  pattern  found  in  people 
who  resent  the  control  of  their  actions  by  any  outside  authority. 
The  general  thought  is  "Do  not  tell  me!  No  one  can  tell  me  what 
to  do.”  A  person  having  this  thougTvt  will  disregard  rulea  and 
procedures  if  they  prevent  him  or  her  from  doing  things  his  or  her 
own  way.  They  might  ignore  a  parking  sign  (Do  not  tell  me  where 
to  park!)  or  not  follow  the  prescribed  preflight  checklist  jDo  not 
tell  me  what  to  do  to  get  my  aircraft  ready  to  fly!).  The  key  to 
the  behavior  is  that  the  person  is  resentful  of  established  rules, 
regulations,  and  procedures,  and  will  tend  to  ignore  advice,  even 
though  it  is  well-founded. 

2.  Impulsivity .  This  is  the  thought  pattern  found  in  people 
who,  "when  facing  a  moment  of  decision,  feel  that  they  must  do 
something  and  do  it  quickly.  This  thought  is  characterized  in  the 
student  manual  as,  "Do  something  -  quickly!"  The  person  having 
this  thought  does  not  stop  to  think  about  what  to  do,  does  not 
explore  the  implications  of  what  he  or  she  is  about  to  do,  or  does 
not  examine  a  set  of  alternatives  and  select  the  best  one.  This 
person  simply  does  the  first  thing  that  comes  to  mind. 

3.  Invulnerability .  This  is  the  thought  pattern  of  people  who 
feel  that  nothing  disastrous  could  happen  to  them.  The  thought  is 
characterized  in  the  student  manual  by  the  statement,  "It  won't 
happen  to  me!"  Serious  illness,  floods,  fires,  and  volcano 
eruptions  are  extremely  remote  to  people  having  this  thought. 
They  know  that  such  disasters  happen,  but  never  feel  that  they 
will  be  directly  affected.  People  who  think  this  way  are  more 
likely  to  take  chances  and  unwise  risks.  They  feel  that  accidents 
will  happen  to  other  people,  but  not  to  them. 
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4.  Macho.  This  is  the  thought  pattern  of  people  who  are  always 
trying  to  prove  that  they  are  better  than  others.  The  thought  is 
characterized  in  the  student  manual  by  the  statement,  "I  can  do 
it!"  They  feel  that  others  should  be  careful,  but  not  themselves. 
They  feel  that  they  can  always  manage  to  handle  the  difficult 
situations.  They  prove  themselves  by  taking  risks  and  try  to 
impress  others  by  acting  dangerously.  While  the  macho  thought 
pattern  is  generally  considered  to  be  a  characteristic  of  male 
thinking,  it  is  not  so  in  this  case.  In  the  sense  used  here, 
macho  is  more  closely  associated  with  overconfidence,  but  goes 
beyond  the  meaning  of  that  word  in  that  the  individual  attempts  to 
do  difficult  things  in  order  to  gain  the  admiration  of  others.  Ir 
this  sense,  the  characteristic  is  not  restricted  to  males. 

5.  External  control.  People  who  have  this  thought  pattern  feel 
that  they  can  do  very  little,  if  anything,  to  control  wha*- 
happens.  This  thought  is  characterized  in  the  student  manual  by 
the  question,  "What's  the  use?"  When  things  go  well,  it  is 
attributed  to  good  luck.  When  things  go  badly,  it  is  attributed 
to  bad  luck,  or  it  is  generally  the  fault  of  someone  else.  People 
having  this  thought  may  at  times  exhibit  a  degree  of  paranoia. 
When  in  school,  for  example,  they  might  attribute  good  grades  to 
an  easy  examination,  but  may  consider  poor  grades  to  be  the  fault 
of  an  unfair  teacher.  Since  they  feel  that  whatever  they  do  makes 
no  difference,  they  tend  to  be  passive.  Since  they  feel  that  they 
can  exercise  little  or  no  control  over  situations,  they  do  not 
make  decisions  themselves.  They  leave  decision-making 
responsibilities  to  others. 

SELF-ASSESSMENT  INVENTORY.  An  assessment  instrument  was  designed 
to  highlight  the  hazardous  thought  patterns  of  the  students. 
While  this  test  is  based  on  real  data,  modification  and  validation 
will  be  needed  before  it  can  serve  as  a  diagnostic  instrument. 
The  administration  of  the  assessment  inventory  to  a  large  number 
of  pilots  would  provide  normative  data  for  use  on  the  profile. 
Such  work  would  be  a  major  undertaking  in  its  own  right.  For  the 
current  program,  the  most  advantageous  use  of  the  assessment 
scales  is  to  motivate  the  student,  help  h.im  to  identify  his  own 
response  tendencies,  and  serve  as  an  introduction  to  the  five 
hazardous  thoughts. 

ANTIDOTE:,  FOR  HAZARDOUS  THOUGHTS.  The  mere  recognition  of 
hazardous  thoughts  was  not  considered  a  sufficient  means  of 
altering  a  pilot's  tendency  to  make  poor  judgments.  In  addition 
to  recognizing  the  presence  of  hazardous  thinking,  pilots  needed 
to  be  taught  to  take  corrective  actions  against  such  thoughts. 
The  judgment  training  materials  teach  students  to  remove  the 
effect  of  a  hazardous  thought  by  substituting  a  good  judgment 
thought  for  the  hazardous  one.  This  set  of  substitute  thoughts  is 
called  tne.  antidotes  for  hazardous  thoughts.  The  value  of  the 
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antidotes  goes  beyond  simple  substitution  for  hazardous  thoughts. 
It  is  a  tenet  of  behavior  modification  that  a  change  of  thoughts 
will  promot  c.  cnange  in  actions.  Thus,  teaching  pilots  to  think 
the  antidote  thoughts  is  an  important  device  in  altering  not  only 
the  poor  judgment  thinking  of  pilots,  but  also  the  actions  which 
result  from  such  thinking. 

IDENTIFYING  AND  REDUCING  STRESS.  An  overstressed  state  of  mind 
can  increase  the  likelihood  of  poor  judgments.  In  order  to  teach 
pilots  to  effectively  cope  with  the  effects  of  too  much  stress,  a 
preliminary  need  was  identified  for  a  means  to  assess  the  amount 
of  stress  being  experienced  at  any  given  time. 

The  subjective  units  of  discomfort  (SUD)  system  is  used  as  a  means 
for  identifying  stress.  It  was  developed  by  Wolpe  (reference  8) 
as  a  subjective  anxiety  scale  to  measure  a  person's  anxiety 
responses  to  specific  situations.  Use  of  the  system  is  an 
important  first  step  in  reducing  stress  in  that  it  provides  the 
student  a  way  of  gauging  anxiety  or  tension.  The  system  is 
introduced  by  having  the  student  think  of  the  situation  which  led 
to  the  worst  anxiety  he  has  experienced,  or  a  situation  which 
could  lead  to  the  worst  anxiety  he  or  she  could  imagine,  and 
assign  to  this  situation  the  number;  100.  Next,  the  student  is 
asked  to  think  of  being  absolutely  calm  and  call  this  state  zero. 
All  other  states  of  anxiety  may  then  be  placed  in  a  relative 
position  on  this  scale.  The  student  is  then  presented  with  a 
number  of  situations,  and  asked  to  rate  the  SUD  level  of  each 
situation.  This  information  is  then  used  by  the  student  in 
developing  a  scale  for  assessing  his  or  her  own  stress  level. 

When  stress  levels  are  identified  as  being  excessive,  the  student 
is  taught  to  employ  a  method  for  reducing  stress.  A  simple  deep 
breath  technique  is  provided  as  the  primary  relaxation  method.  A 
more  elaborate  method  called  "Progressive  Relaxation"  is  also 
provided  as  an  appendix  to  the  lesson  on  stress. 

IMPLEMENTATION. 

Instructions  for  implementing  the  judgment  training  program  are 
contained  in  the  Instructor  Manual,  volume  III  of  this  report. 
However,  it  is  appropriate  to  briefly  discuss  here  the  educational 
theory  underlying  the  implementation  design.  Two  general  learning 
theories  are  used  in  the  judgment  training  program:  behavior 
modification  and  facilitation. 

References  for  behavior  modification  are  numerous.  A  concise 
overview  of  the  process  may  be  found  in  Chaplin  and  Krawiec 
(reference  9).  Briefly,  behavior  modification  is  the  application 
of  principles  of  learning  to  achieve  changes  in  motivation, 
skills  and  performance.  Components  of  behavior  modification 
relevant  to  the  manuals  are  stimulus ,  response  and  reinforcement . 
The  stimulus  is  something  which  causes  a  person  to  take  some  sort 
of  action.  The  action  is  the  response.  If  a  pilot  experiences  an 
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engine  failure  on  takeoff  (the  stimulus)  ,  he  or  she  will  then 
accomplish  the  appropriate  emergency  procedure  (the  response) . 
The  purpose  of  behavior  modification  is  to  train  the  person  to 
make  the  proper  response  through  positive  reinforcement .  If  a 
person  takes  some  sort  of  action,  and  something  pleasant  happens 
to  the  person  as  a  result  of  the  action,  then  that  person  is 
likely  to  repeat  that  action  in  the  future.  Ir.  this  case,  the 
action  is  said  to  be  reinforced .  If  something  unpleasant  happens 
as  a  result  of  some  action,  then  that  action  is  said  to  be 
punished  and  the  person  is  likely  to  cease  taking  the  action.  A 
problem  with  punishment  is  that  it  does  not  encourage  a  person  to 
take  a  proper  action;  it  only  encourages  a  person  to  cease  an 
action,  whether  it  is  proper  or  improper. 

Facilitation,  simply  defined,  is  the  process  by  which  one  person, 
who  is  the  trainer  "A,"  helps  another  person  "B,"  grow  or  learn  in 
directions  which  best  suit  "B"  and  are  chosen  by  "B."  The  classic 
paper  describing  the  conditions  of  facilitation  was  authored  by 
Carl  Rogers  (reference  10).  An  overview  of  facilitation  as  a 
component  in  humanistic  educational  theory  may  be  found  in 
Chapter  2  of  Stanford  and  Roark  (reference  11).  Facilitation 
underlies  an  associated  theory  of  learning  which  affirms  that 
experiential  learning  is  the  area  in  which  significant  change 
occurs.  Experiential  learning  has  the  quality  ^f  personal 
involvement.  A  person's  feelings  are  coupled  with  the  cognitive 
aspects  of  being  in  the  learning  event.  This  results  in  a 
difference  in  the  behavior,  the  attitudes  and,  according  to  Rogers 
(reference  12),  perhaps  even  the  personality  of  the  learner. 

As  was  stated  previously,  extensive  use  was  made  of  behavior 
modification  and  facilitation  in  designing  the  program.  To 
elucidate  how  these  learning  theories  were  employed,  the  following 
paragraphs  quote  the  principles  of  judgment  instruction  from  the 
Instructor  Manual.  Each  principle  is  then  followed  by  a  brief 
statement  about  its  relevancy  to  behavior  modification  or  to 
f aci 1 i tat ion . 

"1.  The  Student  Manual  is  simple  to  understand,  and  it  is 
repetitive.  This  is  for  two  beneficial  reasons;  1)  The 
simplicity  produces  frequent  success  experiences  giving  the 
student  a  continuing  exposure  to  positive  reinforcement;  2)  the 
repetition  builds  good  judgment  habits  and  refreshes  the  memory  so 
that  information  can  be  readily  recalled  in  a  variety  of 
circumstances,  not  just  in  the  context  in  which  it  was  learned.” 

This  13  behavior  modification.  Frequent  positive  reinforcement  is 
a  particularly  strong  behavior  modification  device. 
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"2.  The  Student  Manual  presents  numerous  true  stories  to 
stimulate  the  student's  interest  in  and  appreciation  for  good 
pilot  judgment.  Discussions  of  these  stories  and  similar  ones 
from  the  instructor's  personal  experience  are  important  for 
developing  the  student's  judgment.” 

This  is  facilitation.  The  instructor  is  encouraging  or 
facilitating  the  student  to  make  use  of  information  learned  by  the 
student  from  the  Student  Manual. 

"3.  The  instructor  profoundly  affects  the  student  as  a  role  model 
and  as  an  opinion  shaper.  The  instructor's  attitudes  to  safe 
flying  and  to  the  judgment  training  material  may  influence  the 
student's  judgment  more  than  does  the  content  of  the  flight 
training  program." 

The  student's  exposure  to  the  flight  instructor  is  experiential 
learning,  and  it  is  dependent  upon  the  instructor  being  a  good 
facilitator  of  attitude  development.  Citing  many  case  studies, 
management  consultant  J.  Sterling  Livingston  concludes  that  the 
first  year  is  a  critical  period  of  learning  for  new  trainees 
(reference  13).  He  states  that  a  young  person's  first  manager  is 
likely  to  be  the  most  influential  person  in  his  or  her  career.  If 
this  manager  is  unable  or  unwilling  to  help  the  young  trainee 
develop  skills  necessary  to  perform  effectively,  the  trainee  will 
set  lower  standards  for  himself  than  he  is  capable  of  achieving. 
His  self-image  will  be  impaired,  and  he  is  likely  to  develop 
negative  attitudes  toward  his  job,  his  supervisor,  and  even  his 
career  in  business.  If,  on  the  other  hand,  his  manager  helps  him 
achieve  his  maximum  potential,  the  new  trainee  will  build  the 
foundation  for  a  successful  career.  While  Sterling's  work  focuses 
on  the  industrial  manager,  it  can  be  argued  that  the  flight 
instructor  is  also  a  manayer.  As  such,  the  instructor's 
effectiveness  is  reflected  in  the  student's  performance  and 
self-image . 

"4.  Instruction  is  greatly  improved  when  the  instructor  acts  as  a 
coach  and  consistently  uses  the  principles  of  behavior 
modification . " 

The  Instructor  Manual  makes  an  overt  attempt  in  the  introduction 
to  encourage  the  instructor  to  employ  behavior  modification 
techniques . 

"5.  Use  of  the  special  judgment  concepts  in  conversations  with 
the  student  effectively  focuses  instruction  on  judgment  related 
training,  encourages  proper  use  of  behavior  modification,  and 
increases  the  student's  ability  to  provide  the  self -generated 
feedback  upon  which  good  judgment  depends." 

The  concepts  presented  by  unit  I  are  tools  to  help  the  student  and 
the  instructor  deal  with  judgment  in  the  training  environment. 
Additionally,  they  are  devices  to  motivate  changes  in  specific 
behaviors  through  the  use  of  behavior  modification. 
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"6.  Knowing  how  to  recognise  and  respond  to  hazardous  thinking 
and  high  stress  is  very  important  to  exercising  good  pilot 
judgment.  The  instructor  encourages  the  student  to  develop  these 
skills,  but  never  attempts  to  analyze  or  to  modify  the  student's 
personality. " 

The  encouraging  efforts  of  the  instructor  are  intended  to 
facilitate  the  student’s  positive  experiences  with  identifying  his 
own  hazardous  thinking  and  high  stress  levels. 

"7.  The  student  learns  concepts  and  behavioral  techniques,  then 
repeatedly  applies  this  learning  to  relevant  flight  situations 
during  ground  and  flight  training.  The  five  application  lessons 
and  the  in-flight  experiential  activities  are  purposely  spaced 
throughout  the  standard  private  pilot  training  course  in  order  to 
build  new  behavior  habits  through  repeated  reinforcement  and 
constant  student  involvement.  Having  a  student  merely  'learn 
about'  the  judgment  concepts  and  behavioral  aspects  cannot  be 
expected  to  change  pilot  judgment.  Therefore,  an  intensive 
learning  format  which  teaches  only  the  content  of  the  Student 
Manual  over  a  few  days  is  not  acceptable.  Spaced  practice  that 
includes  repetition  and  feedback  with  positive  reinforcement  is 
essential  to  the  success  of  this  judgment  training  program.11 

This  summary  statement  refers  to  both  behavior  modification  and 
facilitation.  The  repeated  practice  with  feedback  is  a  behavior 
modification  reinforcement  technique.  The  emphasis  on  going 
beyond  merely  learning  about  the  judgment  materials  stems  from 
facilitat  .on.  The  applications  lessons  (Student  Manual  unit  III) 
and  the  two  series  of  in-flight  lessons  (Instructor  Manual 
units  IV  and  V)  are  specifically  designed  to  provide  the 
experiential  opportunities  necessary  for  facilitative  learning  to 
take  place. 


OPTIMIZATION 


The  optimization  effort  was  undertaken  to  accomplish  two 
objectives:  (1)  to  improve  the  text  on  a  lessor.-by-lesson  basis 
to  most  clearly  and  succinctly  accomplish  learning  objectives,  and 
(2)  to  rearrange  the  lesson  sequence  and  sections  of  individual 
lessons  to  produce  the  most  effective  learning  track, 

APPROACH. 

OPTIMIZATION  OF  THE  STUDENT  MANUAL.  The  optimization  effort  was 
conducted  fn  three  stages .  In  the  first  stage,  three  E-RAU 
beginning  level  flight  students  read  through  the  initial  draft  of 
the  Student  Manual  and  completed  all  of  the  exercises  contained 
therein.  While  they  were  completing  the  manual,  the  students  were 
observed  by  Aviation  Research  Center  personnel.  They  were  also 
interviewed  after  completing  each  section  of  the  manual,  and  again 


after  completing  the  entire  manual.  The  purpose  of  the  observation  and 
interviews  was  to  determine  whether  there  were  (1)  sections  of  the 
text  which  the  students  found  to  be  confusing;  (2)  sections  which 
seemed  too  long,  too  elementary  or  too  repetitious;  or  (3) 
sections  which  seemed  to  be  incorrectly  or  poorly  sequenced.  A 
first  revision  of  the  Student  Manual  was  then  made  based  upon 
information  gained  during  this  process. 

In  the  second  stage,  six  E-RAU  flight  students  read  the  lessons 
and  completed  the  exercises  in  the  revised  draft  Student  Manual 
which  resulted  from  stage  one.  As  they  worked  through  the  manual, 
the  students  recorded  the  time  it  took  them  to  complete  each 
lesson.  Also,  they  noted  the  sections  which  they  found  to  be 
confusing,  repetitive,  or  overly  simplistic.  The  Student  Manual 
was  then  revised  a  second  time  based  upon  information  gathered 
during  this  small  group  study. 

The  third  stage  of  optimization  was  conducted  concurrently  with 
the  evaluation  effort.  The  27  subjects  in  the  evaluation 
experimental  group  were  given  instructions  to  make  notes  in  the 
margins  of  their  Student  Manuals  wherever  they  discovered 
something  they  believed  to  be  incorrect,  confusing  or 
inappropriate.  They  were  also  asked  to  offer  their  subjective 
opinions  regarding  the  Student  Manual.  The  notes  and  commentaries 
of  these  27  subjects  were  collated  and  summarized.  This  summary 
was  then  used  to  revise  the  Student  Manual  a  third  and  final  time. 

Since  the  manual  submitted  with  this  report  should  be  like  the 
manual  used  in  the  validation  effort,  substantial  revisions  were 
not  made  during  this  stage. 

OPTIMIZATION  OF  THE  INSTRUCTOR  MANUAL.  Optimization  of  the 
Initial  draft  o?  the  Instructor  Manual  was  accomplished  in  two 
stages.  The  first  stage  was  comprised  of  two  steps.  First,  two 
E-RAU  senior  flight  instructors  read  the  draft  Instructor  Manual 
to  determine  whether  the  instructions  contained  in  the  manual  were 
of  sufficient  detail  to  enable  effective  presentation  of  the 
judgment  lessons.  The  instructors  indicated  unclear  passages, 
confusing  directions,  or  any  other  deficiencies  as  they  read 
through  the  manual.  Also,  the  instructors  made  any  recommenda¬ 
tions  that  they  thought  would  lead  to  general  improvement  in  it. 

The  second  step  involved  making  the  Instructor  Manual  compatible 
with  the  newly  revised  Student  Manual.  The  Student  Manual  was 
reviewed  for  changes  which  required  corresponding  revisions  in  the 
Instructor  Manual  to  bring  the  two  into  agreement.  The  Instructor 
Manual  was  then  revised  based  upon  the  flight  instructor  commen¬ 
tary  and  the  review  of  the  Student  Manual. 

The  second  stage  of  optimizing  the  Instructor  Manual  was  conducted 
concurrently  with  the  evaluation  effort,  and  was  based  on  informa¬ 
tion  gathered  from  the  three  flight  instructors  involved  in  that 
effort.  As  they  conducted  the  training,  the  instructors  made 
notes  on  the  appropriateness  of  the  Instructor  Manual  as  a  guide 
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for  administering  each  judgment  lesson.  Also,  the  instructors 
were  interviewed  periodically  during,  and  immediately  following 
completion  of  the  training  to  gain  their  subjective  opinions 
regarding  validity  of  the  Instructor  Manual.  The  instructors' 
notes  and  their  interview  comments  then  formed  the  basis  for  the 
second  stage  of  optimization.  As  with  the  Student  Manual, 
substantial  changes  were  not  made  during  this  stage  to  avoid 
discrepancies  between  the  manual  evaluated  and  the  manual 
submitted  with  this  report. 

RESULTS. 

The  Student  Manual  underwent  alterations  throughout  its  content 
and  eleven  major  changes  were  incorporated.  The  Instructor  Manual 
also  underwent  major  revisions,  and  seven  major  changes  were 
incorporated . 

The  time  required  for  the  average  student  to  complete  the  manual 
was  verified  to  be  just  over  seven  hours.  This  is  below  the  range 
of  eight  to  ten  hours  that  was  targeted  at  the  outset  of  the 
optimization  effort.  Completion  time  data  collected  during  the 
optimization  effort  can  be  found  in  appendix  A. 


EVALUATION  EXPERIMENT 

METHOD . 

EXPERIMENTAL  SETTING.  The  experiment  was  conducted  using  students 
of  Embry-Riddle  Aeronautical  University  as  subjects.  All 
activities  took  place  on  E-RAU's  campus  at  the  Daytona  Beach 
(Florida)  Regional  Airport.  Judgment  training  was  provided  by  two 
flight  instructors  who  were  employees  of  the  Aviation  Research 
Center.  Subjects  in  the  experimental  group  completed  judgment 
ground  training  as  well  as  judgment  flight  training.  Other 
students  were  used  as  controls  for  both  the  ground  school  and 
flight  portions  of  the  experiment.  The  Cessna  172  aircraft  used 
in  the  experiment  were  furnished  by  the  Aviation  Research  Center. 

EXPERIMENTAL  APPROACH.  The  major  purpose  of  this  study  was  to 
demonstrate  the  e  f  £ectivene6S  of  the  optimized  student  and 
instructor  pilot  judgment  training  materials  in  a  small  scale 
experiment.  The  experimental  approach  (figure  1)  consisted  of 
three  groups  -  an  experimental  (treatment)  group  and  two  control 
groups.  The  experimental  group  received  an  academic  pretest, 
judgment  ground  and  fliynt  training,  an  academic  posttest,  and  an 
observation  flight  at  the  completion  of  the  training  program.  The 
first  control  group  (group  Cf)  was  the  flight  control  group  and 
received  only  the  observation  flight  at  the  end  of  the  experiment. 
The  second  control  group  (group  Ca)  was  the  academic  control  group 
and  received  the  written  pretest  and  written  posttest. 
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FIGURE  1.  EXPERIMENTAL  APPROACH 


Use  of  the  pretest-posttest  control  group  design  outlined  by 
Campbell  and  Stanley  (reference  14)  ,  for  the  academic  portion  of 
the  experiment,  provided  measures  of  the  acquisition  by  the 
experimental  group  of  the  judgment  concepts.  The  design  also 
controlled  for  differences  which  may  have  been  due  to  experiential 
learning  occurring  during  a  student's  regular  course  of  study. 

The  posttest-only  control  group  design  (reference  14)  ,  was  used 
for  the  flight  portion  of  the  experiment  for  three  reasons. 
First,  it  could  be  reasonably  assumed  that  students  in  the 
experimental  and  flight  control  groups  were  equivalent  at  the 
start  of  the  experiment,  as  they  were  all  enrolled  in  the  same 
course  at  E-RAU.  Second,  the  internal  validity  consideration  of 
sensitizing  the  subjects  to  the  treatment  by  giving  them  a 
pre-treatment  observation  flight  was  avoided.  Third,  time  and 
resource  constraints  prohibited  administering  an  additional 
observation  flight  to  54  subjects. 


Flight  training  for  the  experimental  group  consisted  of  three 
flights  during  which  subjects  were  trained  in  the  application  of 
previously  learned  judgment  concepts  in  actual  flight  situations. 
A  description  of  the  flight  training  activities  is  contained  in 
appendix  B.  Subjects  in  groups  E  and  Cf  were  administered  an 
observation  flight  following  the  training.  (See  appendix  C  for  a 
description  of  the  observation  flight.)  Performances  of  the 
subjects  during  the  observation  flight  were  then  compared  to 
determine  whether  any  statistically  significant  differences 
existed  between  the  two  groups  following  judgment  training. 

Ground  training  in  the  judgment  concepts  consisted  of  supervised 
study  utilizing  the  Student  Manual.  Performance  of  the  experi¬ 
mental  group  (E)  was  compared  with  the  performance  of  the  academic 
control  group  (Ca)  both  prior  to,  and  after  training  had  been 
completed.  Performance  was  measured  by  identical  written  pretests 
and  posttests.  (Refer  to  appendix  D  for  a  reproduction  of  the 
test  instrument,  and  refer  to  appendix  E  for  test  answer  key.) 
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The  primary  statistical  test  used  in  comparing  means  and  standard 
deviations  for  the  written  portion  (academic  pretest  and  academic 
posttest)  of  the  judgment  ground  training  was  analysis  of  variance 
(ANOVA) .  The  primary  statistical  test  used  in  comparing  means  and 
standard  deviations  for  the  flight  portion  of  the  judgment 
training  was  one-way  ANOVA.  The  use  of  these  tests  is  described 
below  in  the  Results  section. 

SUBJECTS .  The  81  subjects  were  randomly  selected  from  student 
volunteers  currently  enrolled  in  E-RAU  flight  programs.  Letters 
inviting  students  to  participate  in  the  experiment  were  sent  to 
all  221  students  enrolled  in  the  FA  103-Basic  Flight  course  for 
the  fall  1981  trimester.  Students  in  the  FA  103  course  have  met 
tlie  requirements  for  solo  flight  and  have  completed  the  private 
pilot  ground  school  course  (AS  100-Foundations  of  Aeronautics)  at 
E-RAU. 

The  mean  age  of  the  subjects  completing  the  experiment  was  19.12 
years.  Their  mean  academic  grade  point  average  was  2.96  on  a  four 
point  scale  (4.0  =  "A").  All  but  four  of  the  subjects  completing 
the  experiment  were  males.  Only  two  of  the  subjects  were  not 
American  citizens. 

All  subjects  were  paid  student  wages  for  their  time  during  the 
experiment.  This  pay  totaled  $60.00  for  each  subject  in  group  E 
and  $10.00  for  each  subject  in  groups  Cf  and  Ca.  Those  w.io  were 
involved  in  the  flight  portions  of  the  experiment  were  not  charged 
for  their  flight  time. 

SELECTION  AND  GROUP  ASSIGNMENT.  Of  the  221  students  in  the  target 
population^  TT2  of  the  students  volunteered  for  the  experiment. 
The  subjects  were  randomly  selected  and  assigned  to  one  of  three 
groups  of  27  students  each.  In  order  to  completely  randomize  the 
selection  and  assignment  process,  the  application  forms  of  all 
volunteers  were  assigned  a  number  ir.  no  predetermined  order.  A 
random  number  generator  was  then  used  to  select  students  for  the 
experiment.  The  subject  having  the  number  matching  the  first 
random  number  was  assigned  to  group  E.  The  subject  having  the 
number  matching  the  second  random  number  was  assigned  to  group  Ca; 
the  third  to  group  Cf;  the  fourth  to  group  E,  and  so  on.  The 
selection  process  was  repeated  until  the  required  81  subjects, 
with  ten  alternates,  were  selected. 

ATTRITION .  Of  the  81  students  originally  chosen  for  the  experi¬ 
ment,  there  were  five  who  did  not  start.  They  were  replaced  from 
the  pool  of  alternates  by  random  selection.  This  number  included 
four  students  who  did  not  show  up  for  their  first  scheduled 
activity  and  one  student  who  withdrew  before  his  first  activity 
due  to  a  personal  emergency.  Of  the  27  students  in  the  experi¬ 
mental  group  who  started  the  experiment,  26  completed  their 
assigned  activities.  One  student  was  unable  to  complete  the 
observation  flight  for  personal  reasons.  There  were  24  students 
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in  the  flight  control  group  and  25  students  in  the  academic 
control  group  who  completed  the  experiment.  Attrition  in  these 
two  groups  was  due  to  scheduling  difficulties. 

INSTRUCTORS .  Two  flight  instructors,  both  of  wham  were  employees 
of  the  Aviation  Research  Center,  were  used  to  deliver  both  the 
ground  and  flight  judgment  training.  The  instructors  were  trained 
to  administer  the  program  by  research  center  staff  before  the 
experiment  began.  The  ground  training  of  the  instructors  in 
judgment  concepts  consisted  of  supervised  study  of  the  Student 
Manual  and  the  delivery  methods  contained  in  the  Instructor 
Manual.  Training  instructors  in  how  to  deliver  the  judgment 
program  during  flight  increments  was  also  conducted  by  project 
staff  and  included  at  least  three  practice  flights  with  E-RAU 
students.  The  major  objective  of  this  training  was  to  standardize 
the  instructors'  delivery  of  the  judgment  training  materials.  The 
4  students  used  in  the  instructor  standardization  process  were 
chosen  from  the  original  pool  of  research  volunteers,  but  they  did 
not  participate  as  subjects  in  the  experiment. 

TRAINING  PROGRAM  COMPONENTS.  Subjects  in  the  experimental  group 
participated  In  a  supervised  study  program  using  the  optimized 
Student  Manual  as  a  text.  The  program  was  administered  by  the 
flight  instructors  who  used  the  optimized  Instructor  Manual  as 
their  guide.  The  purpose  of  this  supervision  was  to  assure 
mastery  of  judgment  concepts  and  terminology  by  the  subjects.  The 
terminology  is  considered  an  integral  part  of  the  judgment  program 
in  that  it  promotes  good  judgment  behaviors.  Subjects  had  to 
complete  specific  sections  of  the  Student  Manual  before  flight 
training  lessons  could  be  delivered.  The  training  schedule  for 
both  the  ground  school  and  flight  portions  of  the  experiment  is 
contained  in  appendix  F. 

Judgment  flight  training,  as  administered  to  the  experimen’ 
group,  consisted  of  three  flights  of  approximately  two  hours  each. 
The  subjects  were  assigned  to  a  flight  instructor  who  monitored 
their  progress  in  the  written  (concept)  material  and  delivered 
their  flight  instruction.  The  subjects  remained  with  the  same 

instructor  throughout  the  training  portion  of  the  experiment.  The 
training  flights  were  integrated  into  the  judgment  concept 
training  schedule  to  provide  for  optimum  transfer  of  conceptual 
skills  to  actual  flight  situations.  During  the  flights,  subjects 
were  instructed  in  the  application  of  the  mental  processes  of  the 
safe  flight,  action  ways,  avoidance  of  hazardous  thoughts,  and 
methods  to  break  poor  judgment  chains.  They  were  also  given 

opportunities  to  demonstrate  their  judgment-making  abilities  in 
various  in-flight  judgment  situations  presented  by  the 
instructors.  Each  of  the  flights  consisted  of  approximately  three 
lessons  concentrating  on  the  use  of  specific  judgment  concepts, 
and  between  seven  and  ten  judgment  situations.  Refer  to  the 
Instructor  Manual  for  a  list  of  situations  that  could  be  chosen  by 
the  instructor.  (Appendix  B  includes  a  description  of  a  typical 
judgment  training  flight) . 
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DATA  COLLECTION. 


PRETEST/ POSTTEST .  A  written  pretest  (appendix  D)  was  administered 
to  both  the  experimental  and  academic  control  groups  prior  to  the 
beginning  of  the  judgment  program.  To  ensure  testing  consistency, 
the  same  test  was  used  as  a  posttest  measure  and  administered  to 
both  groups  at  the  conclusion  of  the  judgment  training. 

The  test  consisted  of  the  following  five  sections: 

1.  Judgment  Concepts  -  Nonjargon.  This  section  tested  the 
subject*  s  knowledge  of  the  concepts  contained  in  the  Student 
Manual  without  using  the  specific  terminology  of  the  judgment 
training  program. 

2.  Seer  rio  Analysis.  This  section  tested  the  ability  of  the 
subject  -o  identify  poor  judgment  behaviors  of  a  pilot  after 
reading  a  scenario  describing  that  pilot's  flight. 

3 .  Scenario  Analysis  -  Factor  Importance  and  Certainty  Rating . 
In  this  section,  the  subject  was  asked  to  specify  which  of  the 
factors  in  the  previous  scenarios  was  the  most  important  in 
leading  up  to  the  incident,  and  to  apply  a  rating  to  indicate  how 
sure  he  or  she  was  of  the  answer. 

4.  Action  Ways.  This  section  of  the  test  assessed  the  ability 
of  the  subject  to  identify  the  ways  in  which  pilots  in  various 
situations  implemented  poor  judgment  decisions. 

5.  Judgment  Concepts  -  Specific  Terminology.  This  section 
tested  the  subject's  recall  of  the  terminology  contained  in  the 
judgment  training  program. 

OBSERVATION  FLIGHT.  Objective  measures  of  judgment-making 
behaviors  were  obtained  during  an  observation  flight  administered 
to  groups  Cf  and  E  following  the  latter's  judgment  training. 
During  the  observation  flights,  the  subjects  were  placed  in  20 
judgment  situations.  Their  performances  (poor  or  good  judgment) 
in  response  to  the  situations  were  recorded  by  an  impartial 
observer.  Observaole  critical  behaviors  that  constituted  the 
correct  response  to  each  situation  were  specified  by  experienced 
instructor  pilots.  Failure  of  the  subject  to  demonstrate  the 
required  behavior  was  recorded  as  a  judgment  error.  The  judgment 
situation  checklist,  containing  the  situations  used  in  the 
observation  flight,  is  in  appendix  C, 

In  order  to  place  the  subjects  at  ease  during  the  observation 
flight,  they  were  briefed  that  the  flight  was  being  conducted  to 
observe  instructional  techniques  in  an  experimental  pilot  judgment 
training  program.  The  observer  emphasized  to  all  subjects  that 
they  were  not  being  given  a  "flight  check"  and  that  information 
gained  about  their  flying  performance  would  be  kept  confidential 
from  E-RAU  flight  instructors  and  other  University  academic 
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personnel.  The  subjects  were  also  told  that  for  any  decisions  to 
be  made  during  the  flight,  they  were  to  make  those  decisions  as  if 
they  were  the  pilot-in-command.  The  instructor  would,  at  times, 
give  route  directions  and  suggest  maneuvers  to  be  performed;  but, 
otherwise  the  subject  was  in  charge  of  the  flight.  Subjects  were 
also  briefed  that  in  the  event  of  a  real  emergency,  the  instructor 
would  take  control  of  the  aircraft.  The  subject,  instructor,  and 
observer  used  headsets  to  facilitate  the  observation  process. 

The  two  observers  and  two  instructors  were  standardized  in  the 
observation  flight  procedures  during  two  practice  flights  using 
E-RAU  students  who  were  not  involved  in  the  experiment.  The  two 
flight  instructors  used  in  the  observation  flights  were  the  same 
instructors  who  delivered  the  training.  However,  to  equalize  the 
apprehension  which  may  have  been  experienced  by  subjects  in  the 
flight  control  group  flying  with  an  unfamiliar  instructor, 
subjects  in  the  experimental  group  were  scheduled  to  fly  their 
observation  flight  with  the  other  instructor.  Observers  were 
members  of  the  Aviation  Research  Center  staff.  The  observers  and 
the  flight  instructors  were  unaware  of  the  identity  of  subjects 
with  regard  to  experimental  or  flight  control  group.  A 
description  of  specific  observation  flight  activities  is  contained 
in  appendix  C.  Instructional  techniques  and  procedures  used  in 
judgment  flight  traininq  and  the  observation  flights  were  reviewed 
by  E-RAU  Flight  Standards  personnel  ana  were  found  to  be  within 
the  bounds  of  safe  flight  instructional  procedures.  At  the 
conclusion  of  the  observation  flight,  subjects  in  both  groups 
completed  a  short  interview  which  provided  subjective  data 
regarding  experiment  participation.  The  interviewer  was  an 
Aviation  Research  Center  staff  member  unfamiliar  to  the  subjects. 

RESULTS. 

Two  methods  were  used  to  objectively  evaluate  the  effectiveness  of 
the  judgment  program.  The  first  method  involved  a  written  test  to 
determine  how  well  the  subjects  had  mastered  the  concepts 
contained  in  the  Student  Manual.  The  second  method  consisted  of 
an  observation  flight  during  which  data  were  collected  on  the 
judgment-making  behaviors  of  the  subjects  in  various  situations. 

A  brief  description  of  statistical  analysis  procedures  used  is 
contained  in  appendix  G. 

WRITTEN  TEST.  The  written  test  was  administered  to  all  subjects 
in  group  E  (experimental)  and  group  Ca  (academic  control).  The 
test  was  divided  into  five  sections  which  included  the  knowledge 
of  general  pilot  judgment  concepts  (section  I)  ,  good  or  poor 
judgment  scenario  analysis  (section  II),  importance  and  certainty 
rating  scenarios  (section  III),  action  ways  scenarios  (section 
IV),  and  judgment  training  terminology  (section  V). 

i 

Data  for  each  of  the  five  sections  of  the  written  tests  were  1 

analyzed  using  an  analysis  of  variance  for  a  mixed  design  with  one  ) 

i 

I 
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between  subject  (control  vs.  experimental)  and  one  within  subject 
(pretest  vs.  posttest)  variable.  In  this  design  the  primary 
interest  is  in  the  interaction  between  the  two  variables.  A 
successful  training  program  should  result  in  a  marked  shift  in  the 
experimental  group  from  pretest  to  posttest  with  little  if  any 
improvement  in  the  control  group  ever  the  same  time. 

While  it  is  possible  to  look  at  the  main  effects  of  control  vs. 
experimental  groups  and  pretest  vs.  posttest  scores,  caution  must 
be  used.  The  possibility  exists  with  a  strong  interaction  that 
apparently  significant  differences  in  main  effects  may  really  be 
artifacts  as  an  examination  of  the  means  will  reveal.  The  scores 
are  grouped  into  two  sets,  those  that  account  for  between  subjects 
variance,  that  is  variance  accounted  for  by  group,  and  those  that 
account  for  within  subjects  variance,  that  is,  variance  accounted 
for  by  time  or  pretest/posttest  differences.  In  the  second  set  of 
scores,  the  interaction  effect  (group  x  test)  is  the  factor  in 
which  we  are  most  interested.  The  lowest  line  in  each  set  of 
scores  represents  the  error  variance  due  to  individual  subject 
differences.  ( sub jects-within-groups  and  test  x  subjects-within- 
-groups) . 

Results  for  section  I  of  the  written  test  (general  pilot  judgment 
concepts)  are  displayed  in  tables  1  and  2.  In  table  1  it  can  be 
seen  that  the  means  of  both  groups  increased,  although  the 
difference  in  group  E  was  much  greater.  In  designs  of  this  type 
one  would  expect  the  experimental  group  scores  to  improve  and  the 
control  group  scores  to  remain  the  same.  Some  small,  but  real 
improvement  may  have  occurred  in  both  groups  as  a  result  of 
non-experimental  events  between  the  pretest  and  posttest. 

The  ANOVA  (table  2)  examines  these  results.  The  F  value  for 
groups  of  9.307  is  statistically  significant  (p<0.01)  indicating 
a  difference  between  groups  E  and  Ca.  The  difference  in  scores 
attributed  to  the  effect  of  the  pretest/posttest  is  shown  by  the  F 
value  for  test  (F  =■  58.809)  and  is  statistically  significant 
(p  < 0.001).  The  most  important  F  value  is  the  one  which 
represents  the  interaction  of  the  above  two  factors  (group  and 
test).  The  F  value  for  group  x  test  (F  =  22.481)  is  also 
statistically  significant  (p  <0.001)  . 

Data  for  the  remaining  four  sections  of  the  written  test  were 
analyzed  in  the  same  manner. 


TABLE  1 


MEANS  AND  STANDARD  DEVIATIONS 
FOR  WRITTEN  TEST  SECTION  I 


Group  E 

Group  Ca 

Row  Marginals 

Pretest 

Mean 

4.806 

5.120 

4.964 

Posttest 

Mean 

8.538 

6.000 

7.269 

Column 

Mean 

6.673 

5 . 560 

6.117 

Marginals 

SD 

1.086 

1.495 

N 

26 

25 

TABLE  2.  SUMMARY  OF  ANALYSIS  OF  VARIANCE 

OF  WRITTEN  TEST  RESULTb ,  SECTION  I, 
GROUPS  AND  GROUP  X  TEST 


Source 

df 

MS 

F 

Be tween-Subjects 

Groups 

1 

31.581 

9.30,** 

Subjects-with in-groups 

49 

3.393 

Within-Subjects 

Test 

1 

135.475 

58.809*** 

Group  x  Test 

1 

51.789 

22.481*** 

Test  x  Sub jects-wi thin- 

Groups 

49 

2.304 

Total 

101 

4.931 

**  p  <  0.01 

***p  <0.001 


Results  for  section  II  of  the  written  test  (good  or  poor  judgment 
scenario  analysis)  are  presented  in  tables  3  and  4.  Again  the 
relevant  test  is  the  interaction  effect  of  group  x  test,  and  in 
this  case  is  significant  (p  <0.05). 


TABLE  3.  MEANS  AND  STANDARD  DEVIATIONS  FOR 
WRITTEN  TEST  SECTION  II 


Group  E 

Group  Ca 

Row  Marqinals 

Pretest 

Mean 

5.038 

5.120 

5.079 

Posttest 

Mean 

5.692 

5.040 

5.366 

Column 

Mean 

5.365 

5.080 

5.223 

Marginals 

SD  0.901 

0.976 

1 

N  26 

25 

TABLE  4 .  SUMMARY  OF  ANALYSIS 

OF  VARIANCE  OF 

WRITTEN 

i* 

TEST  RESULTS,  SECTION  II,  GROUPS 

* 

AND  GROUP  X  TEST 

: 

Source  df 

MS 

F 

Be tween- Sub jects 

Groups  1 

2.076 

1.180**** 

Subjects-within-groups  49 

1.760 

7 

Within-Sub jects 

Test  1 

2.098 

2.580**** 

■=. 

Group  x  Test  1 

3.432 

4.218* 

Test  x  Sub jects-within- 

Groups  49 

0.814 

Total  101 

1.324 

l 

****  Ns 

•f 

*  p  <  0.05 

1 


I 
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Data  for  section  III  of  the  written  test  (importance  and  certainty 
rating  scenarios)  are  presented  in  tables  5  and  6.  As  can  be  seen 
by  the  F  value  for  the  interaction  effect  of  group  x  test  (F  ** 
0.003),  the  differences  among  the  groups  were  not  significant.  In 
the  scenario  analysis  portion  of  the  test,  subjects  were  to  rate 
the  importance  of  various  factors  from  a  scenario  on  a  scale  of 
zero  to  ten.  The  answers  were  to  be  compared  to  scores  of  a  panel 
of  12  experts  who  had  rated  the  same  scenario  (See  appendix  H  for 
panel  of  experts  information) .  Items  were  chosen  for  use  on  which 
both  a  high  level  of  importance  and  a  high  level  of  consensus  (as 
indicated  by  a  standard  deviation  of  no  more  than  1.5  in  the 
distribution  of  scores)  were  recorded  by  the  experts. 

TABLE  5.  MEANS  AND  STANDARD  DEVIATIONS 
FOR  WRITTEN  TEST  SECTION  III 


Group  E 

Group  Ca 

Row  Marqinals 

Pretest 

Mean 

9.441 

9.427 

9.434 

Posttest 

Mean 

9.545 

9.536 

9.541 

Column 

Mean 

9.493 

9.481 

9.487 

Marginals 

SD 

0.497 

0.417 

N 

26 

25 

TABLE  6, 

SUMMARY  OF 
OF  WRITTEN 
GROUPS  AND 

ANALYSIS  OF  VARIANCE 

TEST  RESULTS,  SECTION  III, 

GROUP  X  TEST 

Source 

df  MS  F 

Between- Subjects 


Groups 

1 

0.003 

0.008**** 

Sub ject s-wi thin-g roups 

49 

0.422 

With in- Sub jeets 

Test 

1 

0.291 

3.116**** 

Group  x  Test 

Test  x  Sub jects-within- 

1 

0.000 

0.003**** 

Groups 

49 

0.093 

Total 

101 

0.253 

****  NS 
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The  subjects,  when  analyzing  the  same  scenario,  also  indicated 
that  these  same  items  were  extremely  important.  This  situation 
represents  what  may  be  called  the  "ceiling  effect."  That  is,  all 
responses  were  at  the  top  of  the  scale.  The  test,  therefore, 
could  not  discriminate  among  the  subjects  as  to  which  ones  were 
making  the  better  analysis. 

The  results  of  the  factor  importance  rating  and  the  certainty 
scale  rating  were  deemed  of  little  use  in  determining  performance 
of  the  experimental  and  control  groups.  The  twin  problems  of 
little  expert  agreement  on  the  majority  of  the  questions  and  the 
ceiling  effect  that  occurred  on  the  remaining  questions  suggest 
that  more  time  and  effort  are  necessary  to  refine  and  validate 
this  type  of  questioning.  Since  the  certainty  scale  rating 
responses  were  contingent  upon  the  merits  of  the  factor  importance 
rating,  this  section  of  testing  was  also  disregarded.  The 
responses  to  the  certainty  rating  scale  were,  therefore,  not 
analyzed. 

Results  of  the  analysis  of  the  written  test  scores,  section  IV 
(action  ways  scenarios) ,  are  shown  in  tables  7  and  8.  Again,  the 
F  value  for  the  interaction  effect  is  statistically  significant 
(p  <0.01) . 


TABLE  7.  MEANS  AND  STANDARD  DEVIATIONS 


FOR  WRITTEN 

TEST  SECTION 

IV 

Group  E 

Group  Ca 

Row  Marginals 

Pretest 

Mean 

4.538 

4.480 

4.509 

Posttest 

Mean 

6.962 

5.200 

6.081 

Column 

Mean 

5.750 

4.840 

5.295 

Marginals 

SD 

1.227 

0.910 

N 

26 

25 

26 


Between-Subjects 


Groups 

1 

21.108 

8.996** 

Subjects-within-groups 

49 

2.346 

With in -Sub jerts 

Test 

1 

62.954 

40.222*** 

Group  x  Test 

Test  x  Sub jects-within- 

1 

18.483 

11.809** 

Groups 

49 

1.565 

Total 

101 

2.913 

**  p  <  0.01 
***p  <  0.001 


Results  of  the  analysis  of  scores  for  the  written  test,  section  V, 
the  judgment  terminology  test,  are  displayed  in  tables  9  and  10. 
The  F  value  of  341.834  is  extremely  high  due  to  the  criterion 
referenced  nature  of  this  portion  of  the  test.  Subjects  in  the 
experimental  group,  because  they  were  exposed  to  a  substantial 
amount  of  material  which  subjects  in  the  control  group  were  not, 
would  be  expected  to  outperform  the  control  group  subjects  on  such 
a  test  of  specific  "jargon"  material. 


TABLE  9.  MEANS  AND  STANDARD  DEVIATIONS 
FOR  WRITTEN  TEST  SECTION  V 


Group  E 

Group  Ca 

Row  Marqinals 

Pretest 

Mean 

8.038 

6.960 

7.499 

Posttest 

Mean 

32.308 

7.040 

19.674 

Column 

Mean 

20.173 

7.000 

13.567 

Marginals 

SD 

2.458 

2.458 

N 


26 


25 


TABLE  10.  SUMMARY  OF  ANALYSIS  OF  VARIANCE 

OF  WRITTEN  TEST  RESULTS,  SECTION  V, 
GROUPS  AND  GROUP  X  TEST 


Source 

df 

MS 

F 

Between-Sub jects 

Groups 

1 

4423.301 

366.132*** 

Sub jects-w i thin -groups 

49 

12.081 

Within-Sub jects 

Test 

1 

3778.177 

346.371*** 

Group  x  Test 

1 

3728.690 

341.834*** 

Test  x  Sub jects-within- 

Groups 

49 

10.908 

Total 

101 

129.274 

***  p  <  C .001 

In  general,  subjects  in  the  experimental  group  performed 
significantly  better  on  the  written  posttest  than  did  students  in 
the  control  group,  demonstrating  that  these  students  did  learn  the 
materials  covered  in  the  judgment  training  program.  Even  the 
portion  of  the  test  that  avoided  the  specific  terminology  of  the 
program  indicated  greater  improvement  for  the  experimental  group. 
This  pattern  was  repeated  uniformly  in  the  four  sections  where 
interpretation  was  possible. 

OBSERVATION  FLIGHT.  Analysis  of  the  observation  flight  was 
conducted  using  a  one-way  analysis  of  variance.  The  relevant  data 
are  presented  in  tables  11  and  12.  These  data  are  based  on  the 
twenty  situations  described  earlier  and  represent  the  percent  of 
situations  in  which  the  subjects  responded  correctly.  The  scores 
reported  by  the  observer  and  pilot  were  pooled  for  this  analysis 
since  a  correlation  of  0.96  was  obtained  between  them.  The  high 
correlation  is  due  to  the  objective  nature  in  which  the  judgment 
situations  were  constructed  and  observed. 

TABLE  11.  MEANS  AND  STANDARD  DEVIATIONS 
FOR  THE  OBSERVATION  FLIGHT 
(20  SITUATIONS) 


Group  E 

Group  Cf 

Mean 

74.365 

57.521 

SD 

12.409 

10.878 

N 

26 

24 

28 


TABLE  12.  SUMMARY  OF  ANALYSIS  OF  VARIANCE 
OF  OBSERVATION  FLIGHT  RESULTS 
(20  SITUATIONS) 


Source 


df  MS  F 


Between-Sub jects 

Groups  1 

Sub jects-within-groups  48 


3541.056 

136.897 


25.867*** 


Total 


49 


206.370 


**«  p  <0.001 


The  results  of  this  test  were  again  statistically  significant. 
There  was  concern  that  some  experimental  flight  group  subjects  had 
been  exposed  during  their  judgment  training  to  some  of  the  same 
situations  that  were  tested  on  the  observation  flight.  Seven  of 
the  judgment  situations  used  during  the  observation  flight  had 
also  been  used  during  training,  but  had  not  been  seen  by  all 
subjects.  It  can  be  argued  that  the  training  of  specific  judgment 
situations  is  part  of  the  criterion  referenced  nature  of  the 
judgment  training  program,  and  therefore  should  be  included  in  the 
analysis.  However,  in  an  effort  to  eliminate  any  possible  bias, 
the  seven  situations  were  eliminated  and  the  remaining  13 
situations  analyzed.  Results  of  this  analysis  are  reported  in 
tables  13  and  14. 


TABLE  13.  MEANS  AND  STANDARD  DEVIATIONS 
FOR  THE  OBSERVATION  FLIGHT 
(13  SITUATIONS) 


Group  E 


Group  C? 


Mean  69.365  56,125 
SD  14.757  13.332 
N  26  24 
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TABLE  14.  SUMMARY  OF  ANALYSIS  OF  VARIANCE 
OF  OBSERVATION  FLIGHT  RESULTS 
(13  SITUATIONS) 


Source 

df 

MS 

F 

Between-Sub jects 

Groups 

1 

2187.835 

11.017** 

Sub jects-with in-groups 

43 

198.592 

Total 

49 

239.189 

**  p  <0.01 

As  would  be  expected,  the  F  value  is  slightly  smaller  due  to  the 
fewer  measures  involved  in  the  analysis  and  the  removal  of  the 
source  of  variance  of  some  of  the  students  previously  exposed  to 
judgment  flight  situations.  With  these  factors  eliminated  from 
the  analysis,  the  results  were  still  statistically  significant 

(p  <  0.002)  . 

As  another  check  against  possible  bias  due  to  previous  exposure  of 
students  in  the  experimental  group  to  judgment  situations,  a 
correlation  was  calculated  between  the  number  of  situations 
students  had  been  exposed  to  prior  to  the  observation  flight,  and 
their  score  on  the  observation  flight.  This  correlation  (r  *» 
0.017)  was  not  significant,  indicating  very  little  relation 
between  previous  situations  seen  and  score  on  the  observation 
flight. 

An  attempt  was  also  made  to  determine  whether  there  was  a 
correlation  between  the  subjects  cumulative  grade  point  average  in 
college  and  any  of  the  measures  on  either  the  written  or  flight 
observation  flight  tests.  There  was  none. 

SUBJECTIVE  DATA.  The  subjects  in  groups  E  and  Cf  completed  an 
interview  Following  their  observation  flight.  Most  of  the 
interviews  were  done  immediatel  after  the  flight.  In  instances 
where  the  observation  flight  te.  ..inated  after  the  close  of  office 
hours,  the  interviews  were  done  the  following  business  day.  Each 
interview  was  conducted  by  a  member  of  the  Aviation  Research 
Center  staff  who  was  not  previously  involved  with  the  training  or 
observation  of  the  subjects.  All  interviews  were  conducted  in  a 
staff  scientist's  office  at  the  Aviation  Research  Center. 

The  interview  process  produced  four  sets  of  data: 

1.  Demographic  data  for  the  subjects  in  groups  E  and  Cf; 
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2.  Subjective  response  data  from  subjects  in  group  E 
summarizing  their  opinions  about  various  aspects  of  the 
experimental  training  program; 

3.  Subjective  response  data  from  groups  E  and  Cf  summarizing 
their  opinions  about  various  aspects  of  the  observation 
flight;  and, 

4.  Subjective  response  data  from  group  E  subjects  regarding 
personal  attitudes  about  the  perceived  benefits  of  pilot 
judgment  training. 

Demographic  data  described  in  item  1  above  as  well  as  demographic 
data  for  group  Ca  is  reported  in  appendix  I.  The  data  described 
in  items  2  and  3  above  were  collected  by  oral  questioning  of  the 
subjects  by  the  interviewer.  Data  for  items  2  and  3  are 
summarized  in  tables  15,  ]6  and  17  below.  Appendix  I  presents  a 
sample  of  the  interview  form  employed  for  the  questioning  (see 
pag<_  1-5) . 

The  data  regarding  attitudes  about  judgment,  training  benefits, 
described  above  in  item  4,  was  collected  by  having  the  subjects  in 
group  E  fill  out  a  student  attitude  questionnaire  sheet.  This 
data  is  summarized  in  table  18  below.  A  reproduction  of  the 
questionnaire  form  appears  in  appendix  I  on  page  1-8. 


TABLE  15.  GROUP  E  RESPONSES  TO  QUESTIONS 
ABOUT  THE  EXPERIMENTAL  TRAINING 


Question 


V/ as  the  effort  required  to 
complete  the  manual  too  time 
consuming  ? 


Was  the  time  spent  in  judgment 
flight  training? 

a.  too  little? 

b.  too  much? 

c.  about  right? 


How  would  you  like  to  see  the 
program  changed? 


Response  Summaries 
(N  =  26) 


No  subjects  affirmed  that 
Student  Manual  was  too  time 
consuming.  Eight  stated  it 
seemed  too  redundant. 


Too  little  -  3 
Too  much  -  1 
About  right  =  22 


Nine  subjects  suggested  that 
there  be  no  change.  Others 
indicated  minor  changes.  (See 
next  question.) 


(Continued  to  Next  Page) 


TABLE  15.  GROUP  E  RESPONSES  TO  QUESTIONS 
REGARDING  THE  EXPERIMENTAL  TRAINING  (Continued) 


Which  areas  would  you 
emphasize? 

a.  more 

b.  less 

Need  to  be  emphasized  more: 
stress  reduction  =  4 

hazardous  thoughts  =  4 
judgment  situations  =  4 

14  subjects  indicated  that  they 
would  like  to  see  less 
redundancy. 

Has  this  program  had  any 
impact  on  your  judgment-making 
behaviors  thus  far? 

What  were  the  circumstances? 

1  subject  indicated  that  the 
program  had  no  impact  on 
judgment-making  behaviors.  25 
subjects  indicated  that  the 
program  heightened  their 
awareness  and  that  they  began 
to  recognize  poor  judgment 
situations  even  in  flight 
outside  of  the  program. 

Were  you  asked  anything  on  the 
written  test  that  was  not  in 
the  Student  Manual? 

6  subjects  responded  that  there 
was  material  on  the  test  that 
was  not  covered  in  the  manual. 

Did  you  understand  all  of  the 
questions? 

4  subjects  indicated  having  not 
understood  questions  on  the 
test. 

Did  you  have  ample  opportunity 
to  meet  with  your  instructor? 

1  subject  indicated  inadequate 
time  with  the  instructor. 

How  do  you  feel  about  each  of 
the  following  as  they  related 
to  pilot  judgment? 

a.  Hazardous  thoughts  and 
antidotes 

b.  Action  ways 

c.  Mental  processes  of  safe 
flight . 

d.  Subject  areas 

a.  All  subjects  thought  that 
these  related  well  to  pilot 
judgment . 

b.  High  importance  ■  7 

Little  importance  =  3 

Medium  importance  =  16 

c.  All  were  neutral. 

d.  Responded  favorably  =  6 
while  some  found  it  difficult 
to  relate  to  the  judgment 
situations  and  others  thought 
there  was  too  much  emphasis  on 
this . 

Do  you  feel  this  program  will 
be  valuable  to  you  in  your 
future  flying? 

Why? 

All  indicated  that  the  program 
would  be  of  value.  Many  said 
that  their  awareness  was 
increased  along  with  greater 
anticipation.  Others  indicated 
an  improvement  in  their 
attitude. 
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TABLE  16.  GROUP  E  RESPONSES  TO  QUESTIONS 
REGARDING  THE  OBSERVATION  FLIGHT 


Question 

Response  Summary 
<N  =  26) 

Had  you  heard  about  the 
check-flight  before? 

Only  one  subject  indicated 
having  previous  knowledge  about 
the  observation  flight.  What 
was  heard  was  of  the  nature, 
"Watch  out,  it's  a  trick." 

In  situations  where  the 
instructor  indicated  you  made 
a  poor  judgment  did  you  think 
you  were  coerced  or  forced 
into  the  maneuver? 

One  subject  thought  he  was 
coerced.  This  may  be  due  to 
the  unique  role  of  the 
instructor  during  the 
observation  flight. 

Were  there  any  situations  you 
thought  were  unfair  or  overly 
confusing? 

1  subject  was  confused  about 
the  situation  involving  a  turn 
during  a  simulated  engine 
failure. 

Were  there  any  maneuvers  or 
procedures  requested  of  you 
with  which  you  were  not 
familiar? 

4  subjects  were  not  familiar 
with  S-turns.  1  subject  did 
not  understand  the  endurance 
situation.  3  subjects  were  not 
familiar  with  cross  country 
operations . 

TABLE  17.  GROUP  Cf  RESPONSES  TO  QUESTIONS 
ABOUT  THE  OBSERVATION  FLIGHT 


Had  you  heard  about  the 
check-flight  before? 


In  situations  where  the 
instructor  indicated  you  made 
a  poor  judgment  did  you  think 
you  were  coerced  or  forced 
into  the  maneuver? 


Were  there  any  situations  you 
thought  were  unfair  or  overly 
confusing? 


Were  there  any  maneuvers  or 
procedures  requested  of  you 
with  which  you  were  not 
familiar? 


Response  Summary 
(N  =  24) 


Only  1  subject  indicated  having 
heard  about  the  observation 
flight.  What  was  heard  was  of 
the  nature  "Watch  out,  it's  a 
trick." 


4  subjects  indicated  that  they 
thought  they  may  have  been 
coerced  into  a  maneuver.  This 
may  have  been  due  to  the 
unique  role  of  the  instructor 
during  the  observation  flight. 


1  subject  was  confused  about 
entry  to  an  airport  (used  wrong 
runway) .  1  subject  was 

confused  during  the  disturbed 
approach  path. 


4  subjects  were  not  familiar 
with  the  requested  ground 
reference  maneuvers.  1  subject 
was  unfamiliar  with  control 
tower  procedures  and  requested 
base  leg  pattern  entry.  1 
subject  had  not  done  steep 
turns.  1  subject  had  not  seen 
slips  to  landing. 


TABLE  18.  GROUP  E  RESPONSES  TO  JUDGMENT 
TRAINING  ATTITUDE  QUESTIONNAIRE 


Questions 


Responses 


Strongly  Agree  No  Disagree  Strongly 
Agree  Opinion  Disagree 


(1)  In  the  future  I 
am  going  to  pay  more 
attention  to  the 
judgment  concepts 
covered  in  my  own 
flying. 


(2)  This  program  as  a 
whole  is  a  good  idea. 


(3)  I'd  like  to  see 
material  like  this 
included  as  a 
requirement  for  a 
private  pilot  license. 


(4)  The  instructor 
really  kept  me  aware 
of  judgment  factors 
during  the  training 
flights. 


(5)  I  feel  talking 
about  judgment  with 
the  instructor  is  an 
important  part  of 
learning  good 
judgment. 


(61  I  would  have 
learned  just  as  much 
about  judgment  if 
there  were  no  flying. 


(7)  The  course  moved 
too  fast  and  covered 
too  many  concepts. 


(8)  The  stress 
reduction  lesson  was 
helpful . 


4% 

50% 

8% 

77% 

31% 

8% 

DISCUSSION. 


The  data  indicated  statistically  significant  differences  between 
the  performance  of  experimental  and  control  group  subjects.  This 
was  true  both  in  the  acquisition  of  judgment  concepts  and  in  the 
skills  performance  as  measured  during  judgment  flight  situations. 
The  experimental  group  evidenced  an  increase  in  knowledge  of  the 
judgment  concepts  while  the  control  group  showed  no  significant 
change.  Also,  the  experimental  group  evidenced  a  markedly  better 
performance  on  the  post-treatment  judgment  skills  observation 
flight  than  did  the  control  gjoup. 

One  of  the  major  tenets  of  the  program  is  that  students  must 
master  judgment  concepts  before  those  concepts  can  be  applied 
in  judgment  flight  situations.  Concepts  of  the  judgment 
training  were  not  only  taught  during  the  ground  school,  but 
were  constantly  reinforced  during  flight  training  and  during 
preflight  and  postflight  briefings.  Although  it  cannot  be  said 
that  a  direct  causal  relationship  was  proven  to  exist  between  the 
judgment  ground  training  and  the  improvement  of  judgment  in  flight 
as  measured  by  the  observation  flight,  it  appears  evident  that  the 
integrated  ground  training  and  flight  training  did  have  a  positive 
effect  on  subject  performance  in  judgment  flight  situations. 

The  subjects  participating  in  the  experimental  group  held  strongly 
positive  attitudes  about  the  benefits  of  the  judgment  training 
program  at  the  experiment's  conclusion.  A  substantial  majority  of 
the  group  responded  favorably  to  all  the  questions  posed  by  the 
attitude  questionnaire.  They  reported  that:  1)  they  would  pay 
greater  attention  tc  judgment  concepts  in  their  own  future  flying, 
2)  they  believed  the  program  to  be  "a  good  idea,"  and  3)  they 
would  like  to  see  judgment  training  material  be  a  part  of  the 
required  studies  for  a  private  pilot  license. 

LIMITATIONS .  It  is  noted  that  there  are  limitations  to  the 
genera lizability  of  the  judgment  training  program's  potential 
effects  as  indicated  by  these  treatment  results.  First,  the 
subjects  involved  were  a  somewhat  homogenous  and  unique  group. 
They  were  all  closely  grouped  by  age,  the  mean  being  19.5  years 
with  a  range  of  17-23  years.  They  were  all  full-time  college 
students  enrolled  in  various  aviation  related  degree  programs  at 
an  exclusively  aeronautical  university.  They  were  living  in  an 
aviation  oriented  environment  24  hours  a  day,  and  it  may  be 
surmised  that  virtually  all  of  them  possessed  some  motivation  to 
become  professional  members  of  the  aviation  community.  (See 
appendix  I  for  a  summary  of  demographic  data  collected  about  the 
subjects.)  The  question  then  arises  whether  or  not  the  treatment 
would  transfer  to  a  more  heterogeneous  (i.e.,  age,  personality 
type,  environment,  motivating  factors)  group  of  neophyte  flight 
students . 
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Second,  the  training  period  was  of  a  compressed  nature,  lasting 
only  one  month.  This  period  by  no  means  approximates  the  usual 
training  time  required  for  a  student  to  obtain  a  private  pilot 
license.  This  typically  ranges  from  six  to  twelve  months.  Thus 
no  conclusion  can  be  drawn  as  to  whether  or  not  this  judgment 
training  program,  when  integrated  into  the  "normal"  training 
schedule  at  a  fixed-base  operator  (FBO) ,  will  be  as  effective  a3 
in  this  experiment.  In  addition,  the  evaluation  measures  were 
obtained  immediately  after  the  treatment.  The  experiment  did  not 
investigate  the  extent  to  which  such  training  effects  the  judgment 
of  pilots  over  an  extended  period  of  time.  It  remains  for  future 
studies  to  investigate  the  efficacy  of  judgment  training  extended 
throughout  the  typical  student  pilot's  learning  schedule,  and  the 
extent  to  which  judgment  training  concepts  and  application  skills 
are  retained  over  time. 

CONCLUSIONS. 


Results  of  the  experiment  indicated  that  the  implementation  of  the 
optimized  student  and  instructor  manuals  had  a  positive  effect  on 
the  judgment-making  abilities  of  subjects  in  the  experimental 
group,  as  measured  by  an  observation  flight  and  a  written  test.. 
It  is  hypothesized  that  similar  results  would  be  obtained  in  the 
overall  general  aviation  population,  resulting  in  improved  pilot 
judgment-making  abilities. 


ADDITIONAL  EVALUATION  APPROACHES 


Educational  researchers  generally  identify  two  types  of  evaluative 
research  with  regard  to  instructional  programs.  Lehmann  and 
Mehrens  (reference  15)  provide  a  concise  description  of  the  two 
types : 

One  may,  and  should,  evaluate  a  program  at  several 
stages.  TJ  one  evaluates  at  intermediate  stages 
while  changes  can  still  bo  made  in  the  program,  it 
is  called  process  or  formative  evaluation. 
Evaluation  at  the  completion  of  the  program  is 
referred  to  as  aummative  evaluation.  For  example, 
if  one  has  initiated  a  curriculum  or  instructional 
innovation  in  a  school,  a  systematic  attempt 
should  be  made  to  evaluate  each  stage  of  the 
project,  improve  it  as  it  proceeds  on  the  basis  of 
this  evaluation,  and  then,  at  the  completion  of 
the  project,  summat.ive  evaluation  should  be 
conducted.  ...  All  ...  curriculum  innovations 
should  be  systematical ly  evaluated  both  during  the 
program  to  incorporate  improvements,  and  at  the 
end  to  obtain  summative  (or  decisicn-oriented) 
evaluation . 
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The  optimization  of  the  training  materials  and  the  evaluation 
experiment  reported  here  represent  a  very  constrained  effort  at 
both  formative  evaluation  (optimization  of  the  manuals)  and 
summative  evaluation  (the  small-scale  experiment) .  As  noted 
previously,  there  are  major  limitations  to  the  generalizability  of 
the  reported  results  due  to  the  homogeneous  character  of  the 
limited  subject  pool  and  the  severely  condensed  training  time. 

Also,  no  attempt  was  made  to  investigate  effectiveness  of  judgment 
training  over  an  extended  period  of  time. 

The  judgment  training  program  presented  by  these  volumes  therefore 
stands  in  want  of  further  evaluation.  Additional  formative 
evaluation  of  the  program,  when  used  by  a  randomized  grouping  of 
student  pilot  subjects  in  a  typically  structured  flight  training 
program,  may  produce  evidence  dictating  additional  improvements. 

Further,  a  large-scale  summative  evaluation  is  essential  to 
producing  decision-oriented  results  indicating  the  program's 
enduring  benefits  to  the  aviation  community. 

The  evaluation  experiment  reported  here  offers  one  potential 
evaluation  model  for  discovering  the  need  for  further  improvements 
and  for  providing  empirical  information  about  the  program's 
long-term  value.  Two  additional  approaches  to  evaluate  the 
-’ffects  of  this  program  are  suggested.  The  first  is  a  study  of 
_ne  student's  recall  of  the  pertinent  material.  The  second  is  a 
study  of  post-graduation  pilot  behaviors. 

PERTINENT  RECALL  APPROACH. 

A  program  is  effective  only  if  its  effects  are  lasting.  If  after  j 

a  sufficient  interval  of  time  following  the  completion  of  judgment 
training,  the  graduates  are  again  tested  on  the  materials 
presented  in  the  program,  a  measure  of  retention  can  be  obtained. 

However,  the  ability  to  retain  and  recall  material  from  the 
program,  while  a  necessary  feature  of  success,  is  not  a  sufficient 
criterion  of  that  success.  Mere  recall,  at  a  later  date,  does  not 
indicate  how  salient  or  integral  that  training  is  in  the  pilot's 
thinking  process.  Six  months  after  graduation  we  might  ask  a 

pilot  to  name  the  five  hazardous  thoughts,  and  he  or  she  might 
well  be  able  to  do  that.  But  more  elaborate  testing  methodology 
is  needed  to  determine  whether  the  pilot  would  have  recalled  these 
thoughts  without  prompting  or  whether  he  or  she  would  recall  . 

pertinent  information  in  actual  flight  situations. 

ACCIDENT  REPORT  ANALYSIS.  A  series  of  accident  reports  are  j 

prepared.  Participants  in  the  program  are  asked  to  analyze  the 

reports  and  to  describe  what  went  wrong,  how  the  accident  might  i 

have  been  prevented,  etc.  Actual  accident  data  may  be  used,  but 
it  must  be  presented  in  a  format  that  would  allow  the  opportunity 
to  apply  the  concepts  from  the  training  program.  Such  accident 
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report-  analysis  would  probably  best  be  done  at  various  intervals 
over  a  period  of  time.  One  possibility  is  weekly  administrations. 
The  last  of  these  analyses  would  include  a  series  of  increasingly 
specific  questions  designed  to  determine  the  items  of  learning 
from  the  program  which,  to  that  point,  had  remained  latent.  For 
example,  the  pilot  might  be  asked,  "Is  there  anything  you  learned 
in  your  training  which  helps  you  understand  what  contributed  to 
the  cause  of  this  accident?" 

TV  INTERVIEW  OBSERVATION  AND  REACTION.  TV  tapes  are  prepared  in 
which  an  interviewer  separately  Interviews  two  pilots.  The 
"pilots"  may  actually  be  two  well-rehearsed  actors.  In  each  of 
the  tapes,  the  pilots  differ  from  one  another  on  an  important 
variable  in  the  training  program.  For  example,  one  pilot  might 
discuss  an  instance  when  he  decided  to  fly  under  conditions  which 
suggested  that  he  considered  himself  invulnerable  to  mishap.  The 
other  pilot  will  discuss  a  similar  situation  in  which  he  chose  not 
to  fly.  In  all  cases,  the  presentations  are  arranged  to  permit 
control  of  such  extraneous  variables  as  sequence  of  presentation 
and  specific  actors.  The  interview  must  be  carefully  prepared  to 
conceal  the  hazardous  thought  or  other  element  of  judgment 
training  that  is  being  presented  for  testing.  After  seeing  the 
set  of  interviews,  the  test  subjects  are  asked  to  give  their 
reactions  to,  and  impressions  of,  the  two  pilots.  The  way  in 
which  the  participants  respond  becomes  an  indirect  measure  of 
their  views.  A  semantic  differential  rating  form  is  developed  on 
which  the  subject  rates  the  two  pilots.  An  effective  rating  form 
should  result  in  different  ratings  for  the  pilots  displaying  good 
judgment  and  those  displaying  poor  judgment. 

PILOT  BEHAVIOR  APPROACH. 

An  effective  judgment  training  program  should  result  in  different 
behaviors  for  the  graduates  as  compared  to  non-graduates. 
However,  it  will  be  necessary  to  identify  the  behaviors  which  are 
demonstrative  of  good  judgment  prior  to  implementing  the 
evaluation.  A  suggested  approach  is  to  develop  a  number  of 
in-flight  situations  in  which  pilots  are  provided  opportunities  to 
exercise  their  judgment-making  abilities.  Critical  actions,  or 
behavior  indicating  a  good  judgment  response,  are  then  specified 
for  each  judgment  situation.  These  critical  behaviors  must  be 
observable  in  order  to  assess  the  pilot's  judgment-making  ability. 
Failure  of  the  pilot  to  demonstrate  the  critical  behavior 
constitutes  an  error  in  judgment.  Pilot  judgment  is  then  assessed 
by  comparing  overall  error  rates  among  groups.  Three  possible 
means  of  measuring  pilot  judgment  behavior  follow. 

INSTRUCTOR  EVALUATION.  An  objective  rating  form  is  developed  on 
which  qualified  instructors  rate  the  judgment  behaviors  of  student 
pilots.  This  form  would  cover  all  phases  of  judgment  training. 
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Periodically,  these  instructors  are  asked  to  rate  trainees  in  each 
area.  Graduates  of  the  program  could  a)  so  be  rated  by  other 
qualified  persons  at  various  time  intervals.  An  analysis  of  the 
results  should  provide  evidence  about  differing  pilot  behaviors. 

INCIDENT  REPORTS.  °ilots  with  good  judgment  should  have  fewer 
incidents  (accidents  and  near  accidents)  than  those  with  poor 
judgment.  A  record  of  incidents  involving  graduates  could  be 
carefully  kept  for  several  years. 

GENERAL  BEHAVIOR  REPORTS .  A  means  of  periodically  interviewing 
pilots  about  their  flying  habits  and  experiences  to  elicit  a 
retrospective  record  of  their  benavior  is  designed.  The  interview 
is  an  excellent  technique  for  collecting  data  of  this  type.  This 
is  particularly  true  when  making  formative  evaluations. 

GENERAL  CONSIDERATIONS. 

Those  seeking  a  broader  discussion  of  training  and  performance 
evaluation  methodologies  relevant  to  pilot  judgment  may  wish  to 
refer  to  Jensen  and  Benel's  report,  Judgment  Evaluation  and 
Instruction ;  jr.  Civil  Pilot  Training  (reference  2)  ,  in  particular, 
pages  iv,  xi-xiii,  50T~and  76-110. 

Persons  having  3  professional  interest  in  commenting  upon  these  or 
other  evaluation  approaches  are  invited  to  contact  the  FAA  in  this 
regard.  Please  refer  to  the  specific  contact  information  provided 
in  the  preface  to  this  volume  on  page  iv. 
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APPENDIX  A 


COMPLETION  TIME  -  STUDENT  MANUAL 


This  table  summarizes  completion  times  for  experimental  group 
subjects  for  each  of  the  units  in  the  Student  Manual.  Subjects 
were  requested  to  report  the  times  they  spent  in  each  unit  to 
their  instructor.  Data  for  23  subjects  were  complete  and  are 
reported  here. 


Time  Taken  for 

Time  Taken  for 

Time  Taken  for 

Total  Time 

UNIT  I 

UNIT  II 

UNIT  III 

Hrs.  Mins. 

Hrs.  Mins. 

Hrs.  Mins. 

Hrs.  Mins 

APPENDIX  B 


DESCRIPTION  OF  FLIGHT  TRAINING  ACTIVITIES 

This  appendix  contains  material  from  the  Instructor  Manual 
describing  a  sample  judgment  training  flight.  For  an  explanation 
of  the  judgment  terminology  used  in  this  appendix,  see  pages  4 
through  11. 

SAMPLE  TRAINING  MISSION 

This  is  a  sample  judgment  training  mission  profile.  The  concept 
lesson  and  the  behavioral  situations  to  be  administered  have  been 
selected  for  the  flight  and  are  recorded  on  the  instructor's 
checklist.  All  necessary  preparation  has  been  completed  before 
the  student's  expected  arrival  time. 

Concept  lessor,  to  be  administered; 

Transitions  using  flaps 

Judgment  situations  to  be  administered; 

Preflight  -  fuel  selector  screw 
Preflight  -  shoulder  harness 
Enroute  -  VFR  altitude 
Enroute  -  over  water 

Instructor  Preparation; 

Remove  £uel  selector  screw 
Stow  shoulder  harness 

Locate  available  over-water  practice  area 
Locate  area  of  scattered  clouds  at  3,000  to  8,000  feet 


MISSION  NARRATIVE 


The  instructor  informs  the  student  they  will  meet  at  2:00  p.m.  for 
a  training  flight.  The  student  is  told  to  have  the  aircraft  ready 
to  fly.  The  instructor  arrives  at  the  appointed  time  and  verifies 
that  the  student  has  the  aircraft  ready  to  fly.  The  instructor 
intentionally  makes  no  attempt  to  release  the  stowed  shoulder 
harness . 

Preflight  -  Fuel  Selector  Screw 

The  student  advises  the  instructor  that  he  discovered  the  fuel 
selector  screw  missing  during  the  preflight  check.  He  found  it  on 
the  carpet,  and  used  a  screwdriver  from  one  of  the  mechanics  to 
replace  the  screw.  While  buckling  his  seat  belt,  the  instructor 
offers  positive  reinforcement  for  the  student's  act  of  good 
judgment  regarding  the  fuel  selector.  The  instructor  comments, 
"You  were  very  alert  to  fix  the  loose  fuel  selector.  Good  work." 
The  student  also  buckles  his  seat  belt  but  makes  no  attempt  to 
remove  the  stowed  shoulder  harness. 

Preflight  -  Shoulder  Harness 

The  student  continues  with  the  preflight.  At  this  time,  the 
instructor  says  in  a  non-critical  manner,  "I  notice  that  you  don't 
have  your  shoulder  harness  on.  Are  you  aware  of  the  FARs 
concerning  seat  bel*-3?"  The  student  replies  affirmatively.  The 
instructor  then  asks,  "What  type  of  poor  judgment  action  way  does 
this  represent?"  The  student  answers,  "This  was  a  No  Do;  I  should 
have  fastened  my  shoulder  harness,  but  I  did  not  do  it."  The 
instructor  gives  positive  reinforcement  (praise,  smile)  for  this 
correct  poor  judgment  diagnosis,  and  he  adds  "Don't  let  m^  poor 
judgment  influence  yours  -  you  must  always  make  your  own 
judgment. " 

They  both  then  fasten  their  shoulder  harnesses  and  continue  with 
takeoff  preparations.  The  run-up  and  takeoff  proceed  normally. 
The  aircraft  is  in  flight  at  2,000  feet  AGL  in  the  local  practice 
area.  The  instructor  directs  the  student  to  climb  to  4,000  feet 
and  "to  head  out  in  that  direction  for  some  air  work"  (the 
instructor  points  east  toward  the  ocean).  The  student  replies, 
"Sir,  the  FARs  state  that  for  flights  above  3,000  feet  AGL  on  an 
easterly  heading,  we  must  maintain  odd  thousands  plus  500  feet. 
Wouldn't  5,500  be  a  better  choice?"  The  instructor  compliments 
the  student  (positive  reinforcement)  for  recognizing  a  potential 
poor  judgment  situation. 

Transitions  Using  Flaps 

The  concept  lesson  is  then  conducted  by  the  instructor.  It  is 
devoted  to  training  in  transitions  from  cruise  to  minimum 
controllable  flight  using  various  flap  settings.  Emphasis  is 
placed  on  developing  the  student's  learning  of  the  automatic 
reaction  (AR)  mental  processes.  The  instructor  provides  positive 
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reinforcement  when  appropriate,  and  when  an  error  occurs,  points 
out  the  subject  areas  and  action  ways  in  question. 

The  student  is  told  to  establish  slow  flight  at  minimum 
controllable  airspeed  (MCA) ,  which  he  does  by  reducing  power  and 
extending  flaps,  adding  power  as  MCA  is  neared  to  avoid  a  stall. 
The  instructor  queries  the  student,  "you  are  sure  we  are  clear  of 
other  traffic  in  the  area?"  The  student  replies,  "No,  I  should 
have  made  clearing  turns  while  I  was  setting  up  the  aircraft." 
The  instructor  reinforces  this,  "Yes,  that  was  a  No  Do  action. 
You  should  have  been  more  aware  of  your  environment.  What  you  are 
doing  well  however,  is  understanding  and  controlling  the 
pitch-power  relationship."  The  student  replies,  "Yes,  X  feel  I  am 
doing  pretty  well  with  that."  The  instructor  then  says,  “Very 
good.  Now  retract  the  flaps,  maintaining  your  present  airspeed 
and  altitude."  The  student  retracts  the  flaps  and  reduces  power 
as  he  raises  the  aircraft  pitch  attitude.  Pointing  to  the 
altimeter,  which  reads  100  feet  below  target  altitude,  the 
instructor  comments,  "You  should  have  raised  your  pitch  a  little 
more  -  that  was  an  Under  Do  with  regard  to  the  aircraft.  Not  a 
bad  job  though!"  Recovery  from  slow  flight  is  then  initiated,  and 
the  airwork  continues. 

Enroute  -  VFR  altitude 

Over  the  water  the  student  notices  an  increasing  presence  of 
clouds  at  their  current  altitude.  He  says  to  the  instructor, 
"Soon  we  will  be  unable  to  maintain  VFR  at  this  altitude.  Why 
don't  we  descend  to  3,500  to  get  beneath  these  clouds."  "Good 
observation  and  decision,"  the  instructor  replies.  "Your  repeated 
reviewing  of  the  environment  has  helped  you  avoid  getting  into  a 
poor  judgment  situation."  The  student  maintains  3,500  feet  and 
continues  to  head  northwest  away  from  the  shoreline. 

Enroute  -  Over  Water 

The  instructor  queries  the  student,  "As  a  test  of  your  problem 
resolving  abilities  tell  me  what  would  you  do  now  if  you  lost  all 
power  and  needed  to  make  an  immediate  emergency  landing?"  The 
student  thinks  for  a  few  moments,  and  then  answers  "I  might  have 
to  ditch  in  the  water.  I'm  afraid  I  wasn't  paying  attention,  and 
possibly  we  are  beyond  our  power-off  gliding  distance  to  land." 
The  instructor  pursues,  "That's  correct.  In  your  favor,  though, 
you  have  broken  your  poor  judgment  chain  by  correctly  identifying 
your  judgment  error  of  flying  ar  unknown  distance  from  land.  Now 
tell  me,  what  was  the  subject  area  and  the  action  way  for  this 
judgment?" 

The  student  answers,  "Well,  the  subject  area  is  a  combination  of 
aircraft  ard  environment  because  I  didn't  consider  the  limited 
gliding  range  of  my  aircraft  in  relation  to  how  far  I  had 
travelled  over  water  and  away  from  land.  The  action  way  is  Do.  I 
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should  not  have  let  myself  unintentionally  get  that  far  from 
land."  The  instructor  replies,  "Very  good.  You  clearly  under¬ 
stand  your  error.  Let's  call  it  a  day.  Head  us  back  to  the 
airport." 

Debriefing 

On  the  ground,  the  instructor  and  the  student  review  the  flight  in 
a  debriefing  session.  The  instructor  begins  by  saying,  "You  did 
pretty  well  today.  I  planned  five  judgment  training  activities 
for  you,  and  you  handled  three  of  them  correctly  on  your  own:  the 
missing  screw  for  the  fuel  selector,  the  suggestion  to  fly  at  an 
incorrect  altitude,  and  the  encounter  with  clouds  while  VFR.  You 
did  fail  to  make  proper  use  of  the  shoulder  harness,  and  you 
almost  let  yourself  get  too  far  from  land  at  one  point."  The 
instructor  continues  the  discussion  by  describing  how  a  poor 
judgment  chain  could  develop  if  the  pilot  found  himself  over  water 
with  engine  problems  and  could  not  glide  back  to  land. 

The  “transitions  using  flaps"  concept  lesson  is  then  discussed. 
The  student  is  queried  about  which  mental  process  he  was  using. 
"It  seemed  I  was  using  automatic  reaction,"  he  says.  The 
instructor  then  reviews  the  student's  performance  during  the 
exercise,  and  provides  any  useful  advice  for  possible  improve¬ 
ments.  At  the  end  of  the  discussion,  the  instructor  asks,  "Do  you 
remember  having  any  hazardous  thoughts  during  the  flight?"  The 
student  says  "No."  The  instructor  then  asks,  "How  about  your 
stress  level  -  did  it  ever  increase  to  where  you  felt  uneasy  or 
distracted?"  The  student  replies,  "Most  of  the  flight  I  was  calm 
and  doing  fine.  I  did  get  a  bit  rattled  after  I  realized  I  didn't 
know  whether  or  not  I  could  get  back  to  land  in  an  emergency 
situation.  It  didn't  last  very  long,  but  I  did  think  to  relax 
myself  and  to  deal  with  the  situation  at  hand."  The  instructor 
positively  affirms  the  student's  self  awareness  of  a  change  in 
stress  level.  He  then  declares  that  the  flight  has  ended,  and 
they  make  arrangements  for  the  next  training  session. 


APPENDIX  C 


DESCRIPTION  OF  OBSERVATION  FLIGHT 
and 

OBSERVATION  FLIGHT  CHECKLIST 

The  scenario  used  by  the  instructor  pilots  as  a  standardization 
guide  for  the  observation  flights  is  represented  on  the  following 
pages.  Directions  for  the  instructor,  includ  n»,  feedback  points, 
and  a  list  of  the  situations  to  be  administer &'e  included. 

This  appendix  also  reproduces  the  checkli^  .  ’sed  by  both  the 
observer  and  the  instructor  pilot  during  the  observation  flight. 
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description  of  observation  flight 


SCENARIOS 


Flioht  begins  at  Daytona  Beach  Aviation 
ramp.  Instructor  preparation  in  advance 
of  student  arrival  includes:  .  . 

1.  Secure  Jacksonville  sectional 

chart.  . 

2.  Fuel  aircraft  &  prep  f°r  fll9ht. 

3.  NOTAM  and  weather  check  for  nsb  & 
SFB 

4.  Set’  up  pref light  anomalies  for 
test  situation. 

Upon  arrival,  the  student  is  briefed  on: 

P  L  Keeping  the  flight's  activities 

conf iuenti?l .  .  . 

2.  His  or  her  role  as  pilot-in- 
command  for  the  entire 
"Do  what  you  would  normally  do 
using  your  common  sense. 

3  Flight  route:  cross-country 

flight  with  landings  at  NSB,  SFB. 

4  Procedures  in  effect  that  may  be 
unfamiliar  to  E-RAU  students. 

5.  Flight  instructor's  introduction 


of  possible 
situations. 

6.  "Testing  off" 
"My  aircraft," 
gesture  with 
contact. 


poor  judgment 

signals:  verbal 
"I  mean  it,"  and 
hands  or  physical 


DAB-GROUND 

After  briefing,  the  student  ^  given  the 
aircraft  key.  He  or  she  is  told  to  do^a 
complete  preflight  and 

instructor  when  ready  to  fly  w- 

country.  The  student  is  reminded  that  he 
or  sheY  will  be  the  pilot-in-command  for 
the  entire  2-hour  flight. 

The  student  is  expected  either  to  look  at 
a  current  Jacksonville  sectional  char^  to 
airport  information  on  New  Smyrna  Beach 
and  Sanford  or  to  verify  with  the 

instructor  that  the  instructor  will  be 

acting  as  an  information  source  tot :  the 
two  airports.  The  student  does  a  complete 
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SITUATIONS 


1.  Flight  Planning 


-i.  —m 


pre flight  and  must  discover  and  correct  2. 

all  three  anomalies  which  were  pre¬ 
arranged.  The  student  is  debriefed  on  the 
preflight. 

The  "Entire  Flight-Checklist"  situation  is 
observed  by  the  instructor  continuously  3. 
until  termination  of  the  flight  at  DAB. 

The  "taxi-controls  for  wind"  situation  is 
given  at  DAB  while  taxiing  to  runway.  4. 

Instructor  does  debriefing  on  taxi 
exercise  before  allowing  student  to  radio 
for  takeoff  clearance. 

DAB-NSB  EN  ROUTE 

Flight  proceeds  via  either  of  the  south 
departure  routes  from  DAB.  En  Route  5. 

traffic  avoidance  is  observed  as  the 
student  navigates  the  frequently  used  low 
altitude  airspace  along  the  river.  The 
student  is  told  to  make  a  landing  at  New 
Smyrna  Beach  as  the  flight  passes  south  oT  6. 
the  Port  Orange  bridge .  C  ommunication 

awareness  is  noted  as  the  student  sets  up 
for  the  approach  to  NSB. 

NSB-LANDINGS 

In  the  traffic  pattern,  the  "Radio  7, 

Reports"  situation  is  observed.  After 
the  landing,  the  student  is  directed  to 
taxi  on  or  across  another  runway  to  allow 
the  l,Taxi  Across  Runway*  situation  to  be  8. 
observed.  A  feedback  briefing  for 
situations  5,  6,  7,  and  8  is  given  before 
takeoff , 

NSB  LOCAL 

On  takeoff,  engine  failure  is  simulated  at  9. 
500  feet.  Another  landing  approach  is 
made  where  the  instructor  sets  up  a  high  10. 
approach  situation  by  taking  command  and 
turning  base  leg  abeam  the  runway 
threshold.  A  go-around  is  then  made. 
Departure  from  the  airport  is  made  to  the 
southeast  to  take  the  aircraft  out  over 
the  ocean.  The  student  is  told  to  set  up  11. 
for  slow  flight  and  is  observed  traveling 
too  far  from  land.  (Instructor  directs  a 
turn  back  toward  shore  before  aircraft 
does  go  beyond  a  safe  gliding  distance.) 


Preflight  - 
through 
**  FEEDBACK 


Checklist  - 
Entire 


Taxi  -  Controls 
for  Wind 


Traffic 

Avoidance 


Communications 

Awareness 


Radio  Reports 


Taxi  Across 

Runway 

**  FEEDBACK 


Engine  Failure 
**  FEEDBACK 
High  Landing 
Approach 
**  FEEDBACK 


Over  water 

flight 

**  FEEDBACK 
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Aircraft  is  headed  southwest  toward  the 
Edgewater  area  where  ground  reference 
maneuvers  at  800  feet  are  suggested  over 
the  congested  area  near  Massey  Ranch  12. 
residential  area. 

NSB/SFB  EN  ROUTE 

The  student  is  directed  to  head  southwest 
to  Sanford  using  the  lakes  as  visual 
references.  The  student  is  told  to  climb 
to  "cruise  altitude.*1  After  cruise  13. 
altitude  is  established  and  feedback  is 
given,  the  student  is  told  to  plan  for  a 
landing  at  SFB.  As  a  distraction  from  the 
evaluation  activities,  the  student  is  told 
to  practice  finding  ground  reference 
points  by  looking  for  the  fire  tower  or 
power  lines  between  NSB  and  SFB. 

SFB- LANDINGS 

Contact  is  made  with  Sanford  Tower,  and  14. 
the  "Landing  -  Collision  Avoidance" 
situation  observed  before  the  full  stop 
landing. 

Debriefing  on  13  &  14  is  given  before 

takeoff.  A  slip  demonstration  -  down  to 
100  feet,  one  wing-span  from  the  runway 
edge,  sets  up  the  "Disturbed  Approach"  15. 
situation.  The  student  should  execute  a 
go-around.  A  030°  departure  heading  is 
requested  from  the  tower. 

SFB-DAB  EH  ROUTE 

The  flight  continues  at  1500  feet  past  the 
undeveloped  subdivisions  until  the  north-  16 . 

south  power  lines.  S-turns  are  directed 
(3  sets)  then  steep  power  turns  while  at 
800  feet.  An  engine  failure  is  simulated  17. 
(throttle  reduction) ,  timed  to  make  the 
field  northwest  (with  powerlines)  the  best 
choice.  The  need  for  a  90°  turn  is  18. 
simulated  at  500  feet. 

During  recovery,  the  location  and  19. 
endurance  situations  are  given,  followed 
by  feedback  for  16,  17,  18,  19,  and  20  20. 

during  the  flight  back  to  Daytona. 


Congested  Area 
Maneuvers 
**  FEEDBACK 


En  Route-Altitude 
**  FEEDBACK 


Landing-Collision 
Avoidance 
**  FEEDBACK 


Disturbed 
Approach 
**  FEEDBACK 


Low  Altitude 
Airwork 

En  Route-Engine 
Fai lure 

Low-Speed  Turn 


En  Route-Location 

En  Route-Endurance 
**  FEEDBACK 
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CHECKLIST:  JUDGMENT  SITUATIONS 


Student 


Instructor 


Evaluator 


APPENDIX  D 


WRITTEN  PRETEST/ POSTTEST 

JUDGMENT  TRAINING  MATERIALS  TEST 

SECTION  I 

1.  What  three  general  areas  should  be  of  concern  to  the  pilot  who 
is  using  good  judgment?  Check  one. 

_  Lift,  Drag,  Thrust 

_  Airspeed,  Groundspeed,  ETA 

_  Pilot,  Aircraft,  Environment 

_  Weight,  Weather,  Fuel 

2.  After  a  pilot  makes  a  poor  judgment,  are  the  chances  higher  or 
lower  that  he  will  make  another  poor  judgment? 


3.  What  two  sources  of  information  can  a  pilot  use  to  recognize 
poor  judgment? 


4.  As  a  series  of  poor  judgments  increases  in  length,  are  the 
chances  of  continued  cafe  flight  increased  or  decreased? 


5 .  Are  there  things  a  pilot  can  do  to  reduce  stress  while  flying 
an  aircraft? 


6.  Are  there  methods  a  pilot  can  use  to  decrease  the  chance  of 
making  a  series  of  poor  judgments  while  flying? 


7.  Read  the  following  paragraph  and  circle  the  letter  of  the 
answer  which  best  indicates  how  the  pilot  would  react  if  he 
were  thinking  "Do  something  -  quickly!" 

The  pilot  did  not  calculate  fuel  consumption 
correctly  and  did  not  bother  to  "top  off"  the 
tanks  during  his  last  stop.  With  15  minutes 
of  fuel  left,  he  can  make  an  emergency  landing 
on  an  abandoned  dirt  road  just  below.  A 
second  option  is  to  fly  to  an  airfield  which 
he  thinks  is  ten  minutes  away. 

A.  He  turns  toward  the  airfield  and  looks  for  his  sectional 
chart  of  the  area. 

B.  ile  reduces  his  speed  and  altitude  and  looks  for  another 
emergency  landing  site. 

C.  He  radios  for  assistance. 

D.  He  lands  on  the  dirt  road  with  no  further  consideration  of 
other  alternatives. 


8.  Read  the  following  paragraph  and  circle  that  letter  of  the 
answer  which  best  indicates  how  the  pilot  would  react  if  he 
were  thinking  "I  can  do  it  -  I'm  a  good  pilot." 

This  non-instrument  rated  pilot  is  on  a 
pleasure  flight  with  three  friends.  He  is 
advised  that  weather  conditions  are  poor  in 
the  area  where  he  is  headed  due  to  low  clouds 
and  showers.  The  excursion  plan  is  to  view 
some  lakes  and  mountains  in  the  vicinity. 

A.  The  pilot  goes  to  the  area  and  dodges  in  and  out  of  the 
clouds  to  give  his  passengers  a  view  of  the  lakes. 

B.  The  pilot  decides  to  fly  to  the  area  anyway,  figuring  if 
his  time  is  up,  his  time  is  up,  no  matter  what  the 
weather . 

C  The  pilot  changes  plans  and  flies  to  another  area  where  he 
hopes  weather  conditions  are  better. 

D.  Tha  pilot  disregards  the  weather  advisory  and  flies  to  the 
edge  of  the  mountains  to  see  if  he  can  avoid  the  weather. 
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9.  Read  the  following  paragraph  and  circle  the  letter  of  the 
answer  which  best  indicates  how  the  pilot  would  react  if  he 
were  thinking,  "Don't  tell  me." 

The  pilot  does  not  conduct  a  thorough 
preflight  check.  On  takeoff  he  notices  that 
his  airspeed  indicator  is  not  working. 

Everything  else  seems  normal.  His  friend, 
also  a  pilot,  feels  strongly  that  they  should 
discontinue  the  flight  and  return  to  the 
airfield.  The  pilot  then  becomes  upset  with 
his  friend. 

A.  The  pilot  starts  banging  the  indicator  to  get  it  working. 

B.  The  pilot  tells  his  friend  that  it  it  okay  to  fly  the 
plane  anyway  because  the  FARs  are  just  rulus  to  discipline 
pilots . 

C.  The  pilot  tells  his  friend  that  nothir.,  -■  .1  go  wrong  on 

the  flight. 

D.  The  pilot  continues  to  become  upset,  bc(  he  does  nothing 
because  he  feels  there  is  no  use  trying  to  calm  down  his 
friend. 


10.  Read  the  following  paragraph  and  circle  the  letter  of  the 
answer  which  best  indicates  how  the  pilot  would  react  if  he 
were  thinking  "It  won’t  happen  to  me." 

On  a  pleasure  flight,  the  pilot  is  showing  his 
family  some  of  the  local  sights.  His  eight- 
year-old  son  suggests  that  it  would  be  great 
fun  to  fly  under  the  large  bridge  ahead.  His 
wife  is  upset  at  the  idea,  but  his  older  son 
(age  16)  dares  him  to  do  it. 

A.  The  pilot  decides  to  do  it  since  he  thinks  it  is  really 
not  all  that  ricky. 

B.  The  pilot  quickly  pulls  the  aircraft  into  a  steep  climbing 
turn  to  make  the  children  forget  about  the  bridge. 

C.  The  pilot  explains  that  it  is  illegal  to  do  such  a  thing, 
but  he  does  it  anyway  since  no  one  is  around  to  see. 

D.  The  pilot  flies  past  the  bridge  to  be  sure  he  can  do  it, 
then  he  flies  under  to  show  his  family  how  confident  he  is 
of  his  flying  skills. 


SECTION  II 


Bead  the  following  pilot  report.  The  report  is  by  a  pilot 
who  got  into  trouble  because  of  his  poor  decision  making  about 
preflight  activities  and  the  operation  of  an  aircraft  system. 

After  reading  the  report  you  will  be  asked  questions 
regarding  the  pilot's  judgment.  The  sentences  are  numbered  for 
your  convenience. 


Pilot 1 s  Report 

(1)  I  taxied  out  of  the  loading  area  about  15  minutes  later 
than  I  had  planned.  (2)  When  I  got  about  1,000  feet  down  the 
taxiway,  the  tower  called  me.  (3)  They  said  that  they  thought 
they  could  see  smoke  coming  from  my  left  wheel  assembly.  (4)  I 
did  not  want  to  stop  and  check  out  the  problem  because  I  was 
determined  to  get  co  Birmingville  on  time  to  impress  my  boss.  (5) 
I  figured  I  she  .d  do  something  right  away  to  get  the  tower  off  my 
back,  so  I  de  ided  to  speed  up.  (6)  I  thought  maybe  the  rush  of 
air  would  blow  away  the  smoke  that  had  attracted  the  tower's 
attention. 

(7)  Then  the  left  wheel  started  binding  up  a  little.  (8)  The 
airplane  was  moving  pretty  fast,  and  I  was  having  a  hard  time 
steering  it  in  a  straight  line.  (9)  The  dim  taxiway  lights  did 
not  help  matters  any.  (10)  Before  I  really  knew  what  was 
happening,  the  left  wheel  was  off  the  taxiway  and  into  the  grass. 
(11)  I  closed  the  throttle  and  tried  to  stop  as  fast  as  I  safely 
could,  but  the  brakes  were  not  working  on  the  left  side.  (12)  By 
then  I  figured  there  was  nothing  I  could  do  to  overcome  this  run 
of  bad  luck,  so  I  figured  I  would  just  bring  the  plane  to  a  stop 
as  best  I  could.  (13)  I  was  so  busy  cursing  my  bad  luck  that.  I 
did  not  notice  the  underground  fuel  system  box  sticking  up  until 
it  was  too  late.  (14)  I  hit  the  box  at  only  about  5  miles  per 
hour,  but  that  was  fast  enough  to  do  substantial  damage  to  the 
landing  gear.  (15)  It  really  makes  me  mad  to  think  that  the  whole 
thing  was  due  to  a  hydraulic  fluid  leak  in  the  brake  system.  (16) 
I  probably  would  have  noticed  it  if  I  had  done  my  usual  preflight 
inspections . 

******************** 


1.  Consider  sentence  4.  Check  the  statement  which  best 
represents  the  pilot's  judgment. 

_  Good  judgment 

_ _ Poor  judgment 

_ ____  No  judgment  in  this  sentence 


Cannot  tell 


Consider  sentence  5.  Check  the  statement  which  best 
represents  the  pilot's  judgment. 

_  Good  judgment 

_ _  Poor  judgment 

_  No  judgment  in  this  sentence 

_  Cannot  tell 

Consider  sentence  6.  Check  the  statement  which  best 
represents  the  pilot's  judgment. 

_  Good  judgment 

_  Poor  judgment 

_  No  judgment  in  this  sentence 

_  Cannot  tell 

Consider  sentence  7 .  Check  the  statement  which  best 
represents  the  pilot's  judgment. 

_  Good  judgment 

_  Poor  judgment 

_  No  judgment  in  this  sentence 

_  Cannot  tell 

Consider  sentence  11.  Check  the  statement  which  best 
represents  the  pilot's  judgment. 

_  Good  judgment 

_  Poor  judgment 

_ _  No  judgment  in  this  sentence 


Cannot  tell 


Consider  sentence  12.  Check  the  statement  which  best 
represents  the  pilot's  judgment. 

_  Good  judgment 

_  Poor  judgment 

_  No  judgment  in  this  sentence 

_  Cannot  tell 

Consider  sentence  13.  Check  the  statement  which  best 
represents  the  pilot's  judgment. 

_  Good  judgment 

_  Poor  judgment 

_  No  judgment  in  this  sentence 

Cannot  tell 
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SECTION  III 


Read  the  pilot  judgment  scenario  below.  After  you  have  finished 
reading,  you  will  be  asked  a  series  of  questions  about  factors 
contributing  to  the  pilot's  judgment.  (The  sentences  are  numbered 
to  assist  you  in  answering  the  questions.) 

Pilot  Judgment  Scenario 

(I)  Stan  Siler  is  a  19  year  old  college  stuaent  at  home 
enjoying  a  summer  break  from  classes.  (2)  Stan  is  a  non¬ 
instrument  rated  private  pilot  with  156  hours  of  flying  time.  (3) 
Stan's  father  owns  a  new  single  engine  aircraft,  and  Stan  has 
flown  it  35  hours  this  summer. 

(4)  Around  4:00  p.m. ,  Stan  receives  a  telephone  call  from  his 
girlfriend  who  is  attending  summer  classes  at  State  University. 
(5)  She  tells  him  there  is  going  to  be  a  party  at  her  sorority 
house  that  night.  (6)  She  asks  if  he  can  get  the  family  airplane 
and  fly  to  Capital  City  to  accompany  her  to  the  party. 

(7)  Stan  has  not  seen  her  for  three  weeks,  and  he  tells  her 
lie  will  come.  (8)  He  asks  her  to  meet  him  at  the  airport  at  7:00 
p.m.  (9)  Before  hanging  up,  Stan  asks  her  how  the  weather 
is  there.  (10)  She  says  the  sun  has  been  out  most  of  the 
afternoon. 

(II)  Arriving  at  Hometown  airport  about  5:30  p.m.,  Stan 
carefully  preflights  the  airplane.  (12)  Since  his  girlfriend  said 
that  the  weather  was  good  at  Capital  City,  and  since  it  is  nice 
and  clear  locally,  he  does  not  check  the  aviation  weather 
forecast.  (13)  Stan  knows  the  120  mile  flight  to  Capital  City 
will  take  just  over  one  hour.  (14)  At  5:45  p.m.,  Stan  takes  off 
with  2-H  hours  of  fuel  onboard. 

(15)  About  50  miles  from  Capital  City,  Stan  sees  thick  clouds 
ahead  and  to  the  east.  (16)  About  35  miles  out  of  Capital  City, 
Stan  finds  he  is  encountering  an  increasing  amount  of  clouds  at 
his  cruising  altitude  of  3,000  feet.  (17)  Stan  continues  on 
course,  but  he  reduces  his  altitude  to  about  1,500  feet  to  stay 
below  the  clouds.  (18)  Ten  minutes  later  Stan  is  once  again 
dodging  clouds,  and  he  is  concerned  enough  to  radio  the  Capital 
City  FSS .  (19)  He  is  told  that  the  airport  has  recently  gone 
below  VFR  minimums.  (20)  He  is  directed  to  a  small  airport  about 
15  miles  west  of  his  current  position  for  an  alternate  landing. 

(21)  Stan  now  reduces  his  altitude  to  600  feet  to  stay  out 
of  the  clouds,  and  he  knows  he  must  maintain  at  least  500  feet  to 
stay  above  local  hills.  (22)  With  clouds  closing  in  on  all 
sides,  Stan  considers  making  an  immediate  forced  landing  rather 
than  continuing  on  another  10  miles  or  so  in  search  of  the 
unfamiliar  alternate  airport. 
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(23)  Seeing  a  long,  level  field  below  him,  Stan  flies  over  it 
once  to  check  for  obstacles.  (24)  Finding  it  clear,  Stan  executes 
an  emergency  landing  using  proper  procedures.  (25)  Stan  makes  a 
good  landing,  but  the  heavy  rains  have  made  the  recently 
cultivated  field  very  muddy.  (26)  Shortly  after  the  wheels 
encounter  the  soft  mud,  the  aircraft  noses  over. 

******************** 


Questions 


Use  the  two  following  scale  systems  to  answer  all  of  the 
questions. 

A.  Importance  Scale:  What  level  of  importance  would  you 
rate  each  of  the  factors  listed  in  the  question  if  you  were  in  the 
same  situation  as  the  pilot  in  the  scenairo?  Use  a  scale  of  0  to 
10:  10  =  utmost  importance,  0  *  absolutely  no  importance.  You 
may  use  the  same  scale  number  more  than  once . 

B.  Certainty  Scale:  How  certain  are  you  of  the  importance 
rating  you  gave  to  the  factor  listed  to  the  left?  Use  this  scale: 
3  =  absolutely  certain;  2  =  pretty  sure;  1  =  unsure;  0  =  just 
guessing. 

Question  Series  1. 

Look  at  sentence  12,  and  consider  Stan's  situation.  Use  the 
Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors : 

Importance  Certainty 


a.  Weather  information  at  airport 

b.  Time  the  party  starts 

c.  Experience  in  this  type  aircraft 

d.  Weather  information  from  girlfriend 

e.  Type  of  aircraft 
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Question  Series  2, 

Look  at  sentence  17,  and  consider  Stan's  situation.  Use  the 
Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors : 

Importance  Certainty 


a.  Flying  time  to  destination 

b.  Type  of  aircraft 

c.  Cloud  cover  in  area 

d.  VFR  flight  rules 

e.  Reason  for  trip 


Question  Series  3. 

Look  at  sentence  19,  and  consider  Stan's  situation.  Use  the 
Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors : 

Importance  Certainty 


a.  Area  weather  conditions 

b.  Fuel  on  board 

c.  Distance  from  destination 

d.  Time  of  day 

e.  Radio  contact  currently  available 
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Question  Series  4. 

Look  at  sentence  22,  and  consider  Stan's  situation.  Use  the 
Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors; 

Importance  Certainty 

a.  Pilot's  total  flying  time  _  _ 

b.  Area  weather  conditions  _  _ 

c.  Time  of  day  _  _ 

d.  Location  of  alternate  airport  _  _ 

e.  Regulations  for  VTR  flight  _  _ 

f.  Estimated  arrival  time  _  _ 

g.  Type  of  aircraft  _  _ 

h.  Condition  and  contour  of  local 

terrain  _  _ 

i. 


3- 


Capital  City  ATIS  information 
Instruments  on  board  aircraft 


Pilot  Judgment  Scenario  No.  II. 


(1)  The  pilot  was  on  a  cross-country  flight  in  a  high 
performance,  single-engine  airplane.  (2)  The  weather  was  VFR,  and 
he  was  cruising  at  5,500  MSL.  (3)  The  ground  below  consisted  of 
rolling,  tree-covered  hills  interspaced  with  what  seemed  to  be 
grassy  clearings.  (4)  All  appeared  well  until  the  fuel  pressure 
dropped,  and  engine  power  decreased  suddenly.  (5)  The  pilot 
immediately  switched  fuel  tanks  and  turned  on  the  auxiliary  boost 
pumps.  (6)  The  engine  regained  power,  and  the  pilot  climbed  to  a 
higher  altitude. 

(7)  As  the  pilot  applied  climb  power,  the  engine  once  again 
began  to  malfunction.  (8)  This  time  the  power  loss  was  complete. 
(9)  A  minute  passed  before  the  pilot  realized  that  there  was  not  a 
suitable  landing  area  within  gliding  distance. 

(10)  A  glance  at  the  air  speed  indicator  shocked  him  back  to 
reality.  (11)  The  air  speed  had  already  decreased  to  ten  mph 
below  best  glide  speed,  and  the  VSI  was  showing  a  descent  of  1,100 
fpm  —  500  feet  of  precious  altitude  had  already  been  lost.  (12) 
The  pilot  got  hold  of  himself  and  established  power-off  best  glide 
speed,  selected  a  landing  area,  and  established  a  pattern.  (13) 
He  recalled  his  "simulated"  engine  failures,  but  this  was  for 
reall  (14)  Now,  at  only  4,000  feet  and  with  the  VSI  reading  an 
uncomfortable  750  fpm,  the  pilot  had  roughly  five  minutes  to  go. 

(15)  He  considered  a  landing  in  the  tops  of  the  smaller  trees. 

(16)  The  airplane  would  probably  be  heavily  damaged,  but  at  least 
he  would  walk  away.  (17)  He  thought  again  about  the  probable 
damage  and  about  the  possibility  of  never  flying  again.  (18)  No, 
the  tj  ees  were  out  -  he  would  try  for  one  of  the  small  clearings. 
(19)  Still  maintaining  glide  speed  as  he  turned  onto  final 
approach,  it  became  apparent  that  he  would  clear  the  trees.  (20) 
The  pilot  lowered  the  landing  gear  and  dove  the  aircraft  toward 
the  clearing.  (21)  The  aircraft  picked  up  speed  and  struck  the 
ground  hard  on  the  main  gear,  tearing  it  away.  (22)  What  had 
appeared  to  be  a  relatively  smooth  surface  from  the  air  was 
actually  uneven  and  strewn  with  rocks.  (23)  The  aircraft  slid  to 
a  stop  on  its  belly.  (24)  The  pilot  restrained  only  by  a 
seatbelt,  jacknifed  into  the  control  panel,  sustaining  severe  head 
and  facial  injuries. 

******************** 


Questions 


Use  the  two  following  scale  systems  to  answer  all  of  the 
questions . 

A.  Importance  Scale:  What  level  of  importance  would  you  rate 
each  of  the  factors  listed  in  the  question  if  you  were  in  the  same 
situation  as  the  pilot  in  the  scenario?  Use  a  scale  of  5  to  10 : 
10  =  utmost  importance ,  0  =  absolutely  no  importance.  You  may  use 
the  same  scale  number  more  than  once. 
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B.  Certainty  Scale:  How  certain  are  you  of  the  importance 
rating  you  gave  to  the  factor  listed  to  the  left?  Use  this  scale: 
3  =  absolutely  certain;  2  =  pretty  sure;  1  *  unsure;  0  =  just 
guessing. 

Question  Series  5 

~  Look  at  sentence  5  and  consider  the  pilot's  situation.  Use 
the  Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors: 

Importance  Certainty 

a.  Airspeed  _  _ 

jj.  Carb  Heat  _  _  _ 


c.  Altitude 


d.  Weather 


Question  Series  6 

Look  at  Sentence  9  and  consider  the  pilot's  situation.  Use 
the  Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors: 

Importance  Certainty 

a.  Altitude  _ 


b.  Wind 


c.  Airspeed 

d.  Radio 


e.  Engine  Restart 


Question  Series  7 

Look  at  Sentence  12  and  consider  the  pilot's  situation.  Use 
the  Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors: 

Importance  Certainty 

a.  Airspeed  _  _ 


b.  Altitude 


c.  Wind 


d.  Radio 


e.  Terrain 
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Question  Serl.-c  8 

Look  at  Sentence  18  and  consider  the  pilot's  situation.  Use 
the  Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors : 

Importance  Certainty 

a.  Aircraft  Damage  _  _ 

b.  Filing  Accident  Report  _  _ 

c.  Personal  Injury  _  _ 

d.  Airspeed  _  _ 

e.  Altitude  _  _ 

f.  Engine  Restart  _  _ 

g.  Shutdown  Checklist  _  _ 


Question  Series  9 

Look  at  Sentence  20  and  consider  the  pilot's  situation.  Use 
the  Importance  Scale  and  the  Certainty  Scale  to  rate  the  following 
factors: 

Importance  Certainty 


a.  Airspeed 

b.  Altitude 

c.  Aircraft  Damage 

d.  Personal  Injury 

e.  Engine  Restart 

f.  Shutdown  Checklist 
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SECTION  IV 


ACTION  WAYS  QUESTIONS 

1.  In  each  of  the  six  paragraphs  below,  underline  the  phrase 
which  indicates  a  poor  judgment  action  by  the  pilot. 

2.  In  the  right  margin,  briefly  state  why  the  action  indicates 
poor  judgment. 

The  aircraft  was  placarded  against  takeoff  or 
approaches  using  the  auxiliary  fuel  tank.  The 
pilot  took  off  with  the  engine  feeding  from 
the  auxiliary  fuel  tank.  The  engine  failed 
shortly  after  takeoff. 

This  pilot  did  only  part  of  the  recommended 
preflight  checks.  He  attempted  to  abort  his 
takeoff  when  he  noticed  the  controls  were 
binding.  The  pilot  lost  control,  and  the 
airplane  slid  off  the  end  of  the  runway. 

Investigation  revealed  that  the  seat  belt  in 
the  rear  cockpit  was  tied  to  the  control 
stick. 

The  cabin  door  came  open  in  flight,  and  the 
passenger  panicked.  The  pilot  immediately 
decided  he  could  not  correct  the  situation  in 
the  air  and  attempted  an  emergency  landing  on 
a  road.  The  aircraft  slid  off  the  narrow  dirt 
road  into  a  ditch. 

The  pilot  hurried  through  preflight,  yet  he 
did  everything  on  the  checklist.  After  about 
ten  minutes  of  flight  the  pilot  experienced 
partial  loss  of  power  and  made  an  emergency 
landing.  A  careful  inspection  revealed  the 
power  loss  was  due  to  fuel  contamination  and 
water  was  found  in  the  fuel  line. 

The  pilot  was  taking  his  passenger  for  a  low 
flight  over  some  fields  and  was  flying 
downwind.  He  pulled  up  rather  sharply.  The 
combination  of  the  aircraft's  low  altitude  and 
its  relatively  fast  ground  speed  led  the  pilot 
to  believe  that  his  airspeed  was  adequate. 

During  the  sharp  pullup,  the  aircraft  stalled 
and  spun  into  the  ground. 
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The  pilot  made  an  approach  to  the  3200-foot 
landing  strip  in  strong,  gusty  wind.  At  the 
beginning  of  the  flare  a  wind  gust  caused  the 
pilot  to  have  difficulty  in  controlling  the 
aircraft.  After  struggling  to  get  the 
aircraft  settled  down  onto  the  runway,  he 
decided  to  go  around.  The  aircraft  was  unable 
to  gain  sufficient  altitude  to  clear  power 
lines  at  the  departure  end  of  the  runway. 
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SECTION  V 


1.  Can  more  than  one  subject  area  be  involved  in  pilot  judgment? 


2.  Define  Action  Way. 


3.  What  are  the  6  Action  Ways? 


4.  Describe  each  of  the  6  Action  Ways. 


5.  What  is  a  Poor  Judgment  Chain? 


6.  Are  there  steps  a  pilot  can  take  to  break  a  Poor  Judgment 
Chain? 


7.  What  are  the  3  mental  processes  of  safe  flight? 
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8.  Write  the  abbreviations  for  the  3  mental  processes  of  safe 
flight. 


9.  Define  each  of  the  3  mental  processes  of  safe  flight. 


10.  Can  anyone  have  a  hazardous  thought? 


11.  what  are  the  5  hazardous  thoughts? 


12.  The  pilot  does  not  conduct  a  thorough  preflight  check.  On 
takeoff  he  notices  that  his  airspeed  indicator  is  not  working. 
Everything  else  seems  normal.  His  friend  feels  strongly  that 
they  should  discontinue  the  flight  and  return  to  the  airfield. 
The  pilot  then  becomes  upset  with  his  friend.  Which  of  the 
following  alternatives  best  illustrates  the  anti-authority 
reaction? 

a.  The  pilot  tells  off.  the  friend  for  butting  in. 

h.  The  pilot  starts  banging  the  indicator  to  get  it  working. 

c.  The  pilot  tells  his  friend  that  it  is  okay  to  fly  the 
airplane  anyway,  because  the  FARs  are  just  ruies  to 
discipline  pilots. 

d.  The  pilot  tells  the  friend  that  nothing  will  go  wrong  on 
the  flight. 

e.  The  pilot  continues  to  become  upset,  but  he  does  nothing, 
because  he  feels  there  is  no  use  trying  to  calm  down  his 
friend . 
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13.  The  pilot  failed  to  conduct  a  thorough  preflight  check.  On 
takeoff  he  notices  that  his  airspeed  indicator  is  not  working. 
Everything  else  seems  normal.  His  friend  feels  strongly  that 
they  should  discontinue  the  flight  and  return  to  the  airfield. 
The  pilot  then  becomes  upset  with  his  friend.  Which  of  the 
following  alternatives  best  illustrates  the  impulsivity 
reaction? 

a.  The  pilot  tells  off  the  friend  for  butting  in. 

b.  The  pilot  starts  banging  the  indicator  to  get  it  working. 

c.  The  pilot  tells  his  friend  that  it  is  okay  to  fly  the 
airplane  anyway,  because  the  FARs  are  jus*-  rules  to 
discipline  pilots. 

d.  The  pilot  tells  the  friend  that  nothing  will  go  wrong  on 
the  flight. 

e.  The  pilot  continues  to  become  upset,  but  he  does  nothing, 
because  he  feels  there  is  no  use  trying  to  calm  down  his 
friend. 

14.  The  pilot  failed  to  conduct  a  thorough  preflight  check.  On 
takeoff  he  notices  that  his  airspeed  indicator  is  not  working. 
Everything  else  seems  normal.  His  friend  feels  strongly  that 
they  should  discontinue  the  flight  and  return  to  the  airfield. 
The  pilot  then  becomes  upset  with  his  friend.  Which  of  the 
following  alternatives  best  illustrates  the  invulnerability 
reaction? 

a.  The  pilot  tells  off  the  friend  for  butting  in. 

b.  The  pilot  i  irts  banging  the  indicator  to  get  it  working. 

c.  The  pilot  tells  his  friend  that  it  is  okay  to  fly  the 

airplane  anyway,  because  the  FARs  are  just  rules  to 
discipline  pilots, 

d.  The  pilot  tells  the  friend  that  nothing  will  go  wrong  on 
the  flight. 

e.  The  pilot  continues  to  become  upset,  but  he  does  nothing, 

because  he  feels  there  is  no  use  trying  to  calm  down  his 

friend . 
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15.  The  pilot  failed  to  conduct  a  thorough  preflight  check.  On 
takeoff  he  notices  that  his  airspeed  indicator  is  not  working. 
Everything  else  seems  normal.  His  friend  feels  strongly  that 
they  should  discontinue  the  flight  and  return  to  the  airfield. 
The  pilot  then  becomes  upset  with  his  friend.  Which  of  the 
following  alternatives  best  illustrates  the  macho  reaction? 

a.  The  pilot  tells  off  the  friend  for  butting  in. 

b.  The  pilot  starts  banging  the  indicator  to  get  it  working. 

c.  The  pilot  tells  his  friend  that  it  is  okay  to  fly  the 
airplane  anyway,  because  the  FARs  are  just  rules  to 
discipline  pilots. 

d.  The  pilot  tells  the  friend  that  nothing  will  go  wrong  on 
the  flight. 

e.  The  pilot  continues  to  become  upset,  but  he  does  nothing, 

because  he  feels  there  is  no  use  trying  to  calm  down  his 

friend. 

16.  The  pilot  failed  to  conduct  a  thorough  preflight  check.  On 
takeoff  he  notices  that  his  airspeed  indicator  is  not  working. 
Everything  else  seems  normal.  His  friend  feels  strongly  that 
they  should  discontinue  the  flight  and  return  to  the  airfield. 
The  pilot  then  becomes  upset  with  his  friend.  Which  of  the 
following  alternatives  best  illustrates  the  external  control 
reaction? 

a.  The  pilot  telle  off  the  friend  for  butting  in. 

b.  The  pilot  starts  banging  the  indicator  to  get  it  working. 

c.  The  pilot  tells  his  friend  that  it  is  okay  to  fly  the 

airplane  anyway,  because  the  FARs  are  just  rules  to 
discipline  pilots. 

d.  The  pilot  tells  the  friend  that  nothing  will  go  wrong  on 
the  flight. 

e.  The  pilot  continues  to  become  upset,  but  he  does  nothing, 
because  he  feels  there  is  no  use  trying  to  calm  down  his 
friend. 
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Write  the  antidote  for  the  following  thoughts: 


17.  Anti-authority  -  "Don't  tell  me!1 


APPENDIX  E 


PRETEST/POSTTEST  ANSWER  KEY 


SECTION  I  (11  points) 

1. 

Pilot,  Aircraft,  Environ 
ment 

2. 

Higher 

3. 

Self  (1  point) 

Outside  source  (1  point) 

4. 

Decreased 

5. 

Yes 

6. 

Yes 

7. 

D 

8. 

E 

9. 

B 

10. 

A 

SECTION  II  (7  points) 


1. 

Poor  Judgment 

2. 

Poor  Judgment 

3. 

Poor  Judgment 

4  . 

No  Judgment  in  this 

sentence 

5. 

Good  Judgment 

6. 

Poor  Judgment 

7. 

Poor  Judgment 

SECTION  III 

Scores  for  section  III 
were  obtained  by  the  means  of 
student  responses  to  questions 
2C ,  3A,  3B,  4H,  6A,  6C,  6E, 
7A,  7B,  7E,  9A,  9D.  These  are 
the  questions  on  which  the 
experts  had  consensus. 


SECTION  IV  (12  points) 

(Underline  Phrase  =  1  Point) 
(Choose  Action  Way=  1  Point) 


1.  The  pilot  took  off  with 
the  engine  feeding  from  the 
auxiliary  fuel  tank. 

DO 


2.  This  pilot  did  only  part 
of  the  recommended  preflight 
checks. 

NO  DO 


3.  The  pilot  immediately 
decided  he  could  not  correct 
the  situation  in  the  air  and 
attempted  an  emergency  landing 
on  a  road. 

EARLY  DO 


4.  The  pilot  hurried  through 
preflight. 

UNDER  DO 


5.  He  pulled  up  rather 
sharply . 

OVER  DO 


6.  After  struggling  to  get 
the  aircraft  settled  down  onto 
the  runway,  he  decided  to  go 
around . 


LATE  DO 
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SECTION  V  (Max.  =  41) 

1 .  Yes 

2.  Action  which  is  a  result  of  Poor  Judgment 

3.  DO,  NO  DO,  OVER  DO,  UNDER  DO,  EARLY  DO,  LATE  DO 


4 .  DO  - Did  something  you  should  not  have 

OVER  DO  - Did  too  much 

EARLY  DO - Acted  too  early 

NO  DO  - - - Did  not  do  something  you  should  have 

UNDER  DO - Did  not  do  enough 

LATE  DO  - Acted  too  late 


5.  Series  of  errors  in  Judgment 

6 .  Yes 

7.  Automatic  Reaction,  Repeated  Reviewing,  Problem  Resolution 

8.  AR,  RR,  RP 


9. 

Mode  of 

AR  - 

Maintain  ongoing  control  of  the  aircraft 
and  respond  to  emergencies 

Thinking 

PR  - 

Used  to  overcome  undesirable  situations 

that: 

by  means  of  systematic  process 

RR  - 

Keeps  you  constantly  aware  of  all  factors 
that  contribute  to  safe  flight  (P,A,E) 

10.  Yes 

11.  Anti-authority,  Macho,  External  Control,  Invulnerability, 
Impuisivity 

12 .  C 

13 .  B 

14 .  D 

15 .  A 
16  .  E 

17.  Use  the  rules:  they  are  usually  right 

18.  Not  so  fast.  Think  first. 

19.  Why  not  me?  I  am  human,  too, 

20.  Risks  don't  make  my  fly  better.  They  make  me  a  fool. 

21.  I'm  not  helpless.  I  can  make  a  difference. 
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APPENDIX  F 


TRAINING  SCHEDULE  FOR  FLIGHT  PORTIONS 
AND  GROUND  SCHOOL 


This  appendix  represents  the  schedule  form  used  by  the  flight 
instructors  in  the  judgment  program  to  monitor  the  subjects' 
progress.  It  includes  checkpoints  for  work  in  the  Student  Manual 
and  indicates  points  at  which  flight  lessons  are  to  be 
administered.  It  is  to  be  noted  that  the  in-flight  lesson 
schedule  presented  here  was  established  for  use  in  conjunction 
with  the  evaluation  experiment  onl^r.  The  Instructor  Manual 
describes  the  appropriate  scheduling  of  training  activities  for 
use  in  field-implemented  flight  training  programs. 
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APPENDIX  G 


STATISTICAL  ANALYSIS  PROCEDURES 


In  presenting  data  in  this  report,  several  types  of  statistics  are 
used.  Description  statistics  include  the  arithmetic  mean  (M) ,  and 
the  standard  deviation  (SD)  .  The  M  is  commonly  known  as  "the 
average",  and  the  SD  is  a  measure  which  indicates  variability  of 
individual  scores  about  the  M. 

In  evaluating  whether  two  or  more  sets  of  data  (e.g.,  group  E 
pretest,  group  E  posttest,  group  Ca  pretest,  group  Ca  posttest) 
differ  to  a  degree  greater  than  might  be  expected  by  chance  alone, 
statistical  tests  of  significance  are  used.  In  this  report 
"analysis  of  variance  (ANOVA) "  is  the  primary  statistical  test. 

The  amount  of  departure  from  chance  expectation  is  expressed  in 
terms  of  probability  statements.  The  expression  p<  0.05  means 
that  the  probability  that  the  difference  is  due  to  chance  alone  is 
less  than  5  in  100.  The  smaller  the  probability  figure,  the  less 
likely  it  is  that  differences  are  due  to  chance  variation  and  the 
more  likely  it  is  that  the  differences  are  statistically 
significant. 

The  ANOVA  test  yields  a  statistic  called  the  F  ratio,  which  is  the 
ratio  of  two  variance  estimates.  It  is  this  F  ratio  which  allows 
the  probability  statement  to  be  made.  In  the  ANOVA,  reference  is 
made  to  df,  or  degrees  of  freedom.  This  value  refers  to  the 
number  of  independent  measures  on  which  the  test  is  based. 

The  reader  desiring  more  information  on  the  subject  of  statistical 
analysis  is  referred  to  any  one  of  numerous  standard  statistical 
textbooks.  For  example: 


1.  Myers,  J.L.  Fundamentals  of  Experimental  Design.  Allyn  and 
Bacon,  Inc.,  Boston:  1972. 

2.  Runyan,  R.P.,  and  Haber,  A.  Fundamentals  of  Behavioral 
Statistics .  Addison-Wesley ,  Reading,  MA:  1971 . 
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APPENDIX  H 


PANEL  OF  EXPERTS  INFORMATION 


The  panel  of  experts  for  the  scenario  analysis  portion  of  the 
written  test  included  persons  with  the  following  aviation 
backgrounds : 

4  -  test  pilots  from  the  FAA  Technical  Center 

3  -  captains  from  commercial  airlines  with  flight 

instruction  and  safety  supervision  experience 

2  -  flight  instructors  at  major  universities 

1  -  manager  of  safety  for  a  corporate  aircraft  manufacturer 

1  -  editor  of  an  aviation  magazine 

1  -  chief  pilot  for  a  private  corporation 
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APPENDIX  I 


DEMOGRAPHIC  DATA  FOR 
EVALUATION  EXPERIMENT  SUBJECTS 

This  appendix  contains  three  demographic  data  tables,  one  each  for 
the  subjects  in  groups  E,  Cf,  and  Ca.  Data  for  groups  E  and  Cf 
were  obtained  primarily  from  the  interview  after  their  observation 
flight.  All  data  for  group  Ca  and  additional  data  for  groups  E 
and  Cf  were  obtained  from  a  research  subject  information  sheet 
which  was  completed  by  each  subject  at  the  beginning  of  the  study. 

The  appendix  also  contains  the  end  of  experiment  interview  form 
employed  by  the  interviewer  to  record  the  information,  and  it 
contains  the  student  attitude  questionnaire  form  administered  to 
group  E  subjects  during  the  interview. 

REMARKS  REGARDING  INFORMATION  REPORTED. 

Data  for  item  5  on  the  interview  form,  "Time  in  U.S."  (fo*- 
non-U.S.  citizens),  were  not  reported  in  the  tables  to  avoid  the 
possibility  of  identifying  specific  subjects. 

Data  for  item  6  on  the  interview  form,  "Academic  Major,”  were  not 
reported  in  the  tables  because  all  subjects  except  for  one  were 
Aeronautical  Science  majors  (group  E  contained  one  Computer 
Science  major).  At  Embry-Riddle  Aeronautical  University,  the 
Aeronautical  Science  major  includes  studies  in  general  college 
education  subjects  plus  the  operational  and  management  aspects  of 
the  aviation  industry.  In  addition,  all  students  receive  flight 
training  which,  upon  graduation,  qualifies  them  to  be  examined  for 
the  FAA  Commercial  Pilot  Certificate  with  Instrument  and 
Multi-Engine  ratings. 

Flight  hour  data  was  collected  at  beginning  and  ending  points  of 
the  evaluation  experiment.  The  data  reported  in  the  "Flight 
Hours  -  Begin"  column  of  the  tables  were  collected  on  September 
30,  1981  .  The  written  pretests  for  subjects  in  groups  E  and  Ca 
were  administered  on  October  1  and  2,  1981.  The  data  reported  in 
the  "Flight  Hours  -  End"  column  of  the  tables  were  collected  on 
November  6,  1981.  The  written  posttests  were  administered  on 
November  17  and  18,  1981.  Judgement  flight  training  for  the  group 
E  subjects  was  administered  during  the  period  beginning  October  5, 
1981  and  ending  November  6,  1981.  The  observation  flights  for  the 
group  E  and  group  Cf  subjects  were  administered  during  the  period 
November  9,  1981  through  November  25,  1981. 


TABLE  1-2:  DEMOGRAPHIC  DATA:  GROUP  Ca 


i 


Subject 

Citizen- 

Flight  Hours: 

Number 

Age 

Sex 

ship 

GPA* 

Begin 

End 

27 

19 

M 

U.S. 

2.7 

21  .3 

34.9 

28 

26 

M 

U.S. 

2.4 

29.7 

38.4 

29 

19 

M 

U.S. 

2.2 

30.2 

30.2 

30 

20 

M 

U.S. 

2.0 

22.8 

25.7 

31 

20 

M 

U.S. 

3.0 

19.3 

42.6 

32 

18 

M 

* 

3.4 

15.4 

19.0 

33 

18 

M 

* 

3.2 

35.0 

40.9 

34 

21 

M 

U.S. 

2.3 

mmemm 

22.0 

'  35 

19 

M 

U.S. 

3.6 

BtWl  m 

23.6 

36 

19 

M 

U.S. 

2.9 

27.4 

73.4 

37 

19 

M 

U.S. 

3.6 

43.1 

71.0 

38 

18 

M 

U.S. 

2.6 

19.3 

35.9 

39 

20 

M 

U.S. 

2.6 

29.2 

36.2 

40 

20 

M 

U.S. 

3.5 

23.3 

51.8 

41 

19 

F 

U.S. 

1.9 

47.0 

61.5 

42 

21 

M 

U.S. 

2.0 

52.7 

52.7 

43 

24 

M 

U.S. 

3.3 

43.2 

58.1 

44 

17 

F 

* 

2.7 

24.8 

32.6 

45 

20 

M 

^  •  wl  » 

2.7 

19.5 

24.9 

46 

23 

M 

U.S. 

3.0 

20.7 

40.7 

47 

19 

M 

U.S. 

3.0 

25.7 

33.0 

48 

19 

M 

U.S. 

3.3 

25.4 

55.8 

49 

20 

M 

U.S. 

3.2 

41.4 

60.0 

50 

23 

M 

U.S. 

3.5 

91.9 

116.1 

51 

18 

M 

U.S. 

3.4 

27.0 

47.0 

(8% 

(12% 

Mean 

19.96 

Femr.  le) 

Non-US) 

2.88 

30.58 

45.12 

Standard 

Deviation 

2.09 

0.53 

m 

t 


Group  Ca:  N  =  25 

*  =  College  Grade  Point  Average  (4.0  =  A) 
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TABLE  1-3:  DEMOGRAPHIC  DATA:  GROUP  Cf 


Subject 

Citizen- 

mm 

Flight  Hours: 

Number 

Age 

Sex 

ship 

Begin 

End 

52 

19 

M 

U.S. 

3.3 

47.9 

82.7 

53 

19 

M 

U,  S. 

2.5 

30.2 

55.6 

54 

18 

M 

U.S. 

3.6 

73.0 

82.0 

55 

18 

M 

U.S. 

3.1 

22 . 8 

48.4 

56 

20 

M 

* 

2.2 

120.4 

57 

19 

M 

U.S. 

2.5 

50.7 

58 

20 

M 

U.S. 

3.0 

36.4 

36.4 

59 

18 

M 

U.S. 

3.0 

16.0 

28.1 

60 

19 

M 

★ 

3.2 

22.1 

23.5 

61 

19 

M 

U.S. 

3.8 

23.6 

44.2 

62 

20 

M 

U.S. 

2.5 

42.1 

43.7 

63 

19 

M 

U.S. 

2.9 

18.9 

34.5 

64 

18 

M 

U.S. 

3.0 

25.0 

25.0 

65 

18 

M 

U.S. 

2.0 

21.7 

46.8 

66 

21 

M 

U.S. 

3.0 

89.6 

104.9 

67 

19 

M 

U.S. 

4.0 

33.9 

54.6 

68 

23 

M 

U.S. 

2.9 

20.7 

25.9 

69 

20 

M 

U.S. 

2.5 

56.0 

89.0 

70 

18 

M 

U.S. 

2.8 

17 . 3 

17.3 

71 

27 

M 

U.S. 

3.0 

43.6 

44.3 

72 

18 

M 

U.S. 

3.0 

20.3 

20 . 3 

73 

19 

M 

U.S. 

3.1 

19. 4 

52.0 

74 

M 

U.S. 

2.0 

23.5 

27.4 

mm 

M 

U.S. 

mm 

39 . 0 

45 . 0 

Mean 

19.40 

(0  .0% 

Female) 

(8.3% 

Non-US) 

2.90 

37.11 

50.45 

Standard 

Deviation 

2.04 

0.50 

i 

23.11 

28.17 

Group  Cf:  N  =  24 

*  *  College  Grade  Point  Average  (4,0  =  A) 
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END  OF  EXPERIMENT  INTERVI iW 


1 .  NAME _ 

2.  AGE _  3.  SEX _ 

4.  CITIZENSHIP _ 

5.  TIME  IN  U.S. _ 

6.  ACADEMIC  MAJOR _  7.  G.P.A. _ 

QUESTIONS  8-18  ARE  FOR  GROUP  E  ONLY. 

8.  Was  the  effort  required  to  complete  the  manual  too  time 
consuming? 

9.  Was  the  time  spent  in  judgment  flight  training 

a.  too  little? 

b.  too  much? 

c.  about  right? 

10.  How  would  you  like  to  see  the  program  changed? 

11.  Which  areas  would  you  emphasise: 

a.  more? 

b.  less? 

12.  Has  this  program  had  any  impact  or.  your  judgment-making 
behaviors  thus  far?  What  were  the  circumstances? 
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13.  Were  you  asked  anything  on  the  written  test  that  you  were  not 
familiar  with? 

14.  Did  you  understand  all  of  the  questions? 

15.  Did  you  have  ample  opportunity  to  meet  with  your  instructor? 

16.  How  do  you  feel  about  each  of  the  following  as  they  relate  to 
pilot  judgment. 

a.  Hazardous  thoughts  and  antidotes 

b.  Action  ways 

c.  Mental  processes  of  safe  flight 

d.  Subject  areas 

17.  Do  you  feel  this  program  will  be  valuable  to  you  in  your 
future  flying? 

Why? 

18.  Complete  the  attitude  questionnaire. 
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QUESTIONS  19-22  ARE  FOR  GROUPS  E  AND  Cf 


19.  Heard  about  the  check  flight  before? 


20.  in  situations  where  the  IP  (instructor  pilot)  indicated  you 
made  a  poor  judgment  did  you  think  you  were  coerced  or  forced 
into  the  maneuver? 


21.  Were  there  any  situations  you  thought  were  unfair  or  overly 
confusing? 


22.  Were  there  any  maneuvers  or  procedures  requested  of  you  with 
which  you  were  not  familiar? 
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STUDENT  ATTITUDE  QUESTIONNAIRE 


(1) 


(2) 

(3) 


(4) 


(5) 


(6) 


(7) 

(8) 


In  the  future  I  am  going  to  pay 
more  attention  to  the  judgment 
concepts  covered  in  my  own 
flying. 

This  program,  as  a  whole,  is  a 
good  idea. 

I'd  like  to  see  material  like 
this  included  as  a  requirement 
for  a  private  pilot  license. 

The  instructor  pilot  really  kept 
me  aware  of  judgment  factors 
during  the  training  flights. 

I  feel  talking  about  judgment 
with  the  instructor  pilot  is  an 
important  part  of  learning  good 
judgment. 

I  would  have  learned  just  as 
much  about  judgment  if  there 
were  no  flying. 

The  course  moved  too  fast  and 
covered  too  many  concepts. 

The  stress  reduction  lesson  was 
helpful. 


SA 

A 

N 

D 

SD 

********* 

KEY: 

SA:  Strongly  Agree 

A:  Agree 

N:  No  Opinion 

D:  Disagree 
SD:  Strongly  Disagree 
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